Basic Programming Techniques in Python

%load_ext pretty_jupyter

The pretty_jupyter extension is already loaded. To reload it, use:
  %reload_ext pretty_jupyter

Introduction¶

This walk-through is part of the ECLR page.

If you write code for data analysis you will eventually have to master a number of basic programming techniques. They are not directly related to data analysis but will be required for any bit of coding. These are

• Subsetting
• Objects and pointers
• Boolean variables
• if else conditions
• loops
• apply methods
• functions
• lambda functions

In this walkthrough we will show simple examples of these to help you understand code written by other people and to be able to write more complex code yourself.

Preparing your workfile¶

Install the lets-plot package in case you have not yet done so on your computer (pip install lets-plot in your terminal).

We add the basic packages needed for this work:

import numpy as np
import pandas as pd

Loading a dataset¶

Let's get a dataset to look at. We shall use the Baseball wages dataset, mlb1.csv. Either download that file to your working directory or directly link to the file as shown below. Before you upload a csv (or EXCEL) file you should always inspect the file to understand what you should expect. Upon inspection you will find that missing values are coded as "." and hence we pass that information on to the pd.read_csv function.

# Load the dataset
url = "https://raw.githubusercontent.com/datasquad/ECLR/refs/heads/gh-pages/data/mlb1.csv"
df_bball = pd.read_csv(url, na_values = '.')

Let's check out what variables we have in this data-file. Note that Python starts counting variables from 0!

df_bball.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 353 entries, 0 to 352
Data columns (total 47 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   salary    353 non-null    int64  
 1   teamsal   353 non-null    int64  
 2   nl        353 non-null    int64  
 3   years     353 non-null    int64  
 4   games     353 non-null    int64  
 5   atbats    353 non-null    int64  
 6   runs      353 non-null    int64  
 7   hits      353 non-null    int64  
 8   doubles   353 non-null    int64  
 9   triples   353 non-null    int64  
 10  hruns     353 non-null    int64  
 11  rbis      353 non-null    int64  
 12  bavg      353 non-null    int64  
 13  bb        353 non-null    int64  
 14  so        353 non-null    int64  
 15  sbases    353 non-null    int64  
 16  fldperc   353 non-null    int64  
 17  frstbase  353 non-null    int64  
 18  scndbase  353 non-null    int64  
 19  shrtstop  353 non-null    int64  
 20  thrdbase  353 non-null    int64  
 21  outfield  353 non-null    int64  
 22  catcher   353 non-null    int64  
 23  yrsallst  353 non-null    int64  
 24  hispan    353 non-null    int64  
 25  black     353 non-null    int64  
 26  whitepop  330 non-null    float64
 27  blackpop  339 non-null    float64
 28  hisppop   330 non-null    float64
 29  pcinc     353 non-null    int64  
 30  gamesyr   353 non-null    float64
 31  hrunsyr   353 non-null    float64
 32  atbatsyr  353 non-null    float64
 33  allstar   353 non-null    float64
 34  slugavg   353 non-null    float64
 35  rbisyr    353 non-null    float64
 36  sbasesyr  353 non-null    float64
 37  runsyr    353 non-null    float64
 38  percwhte  330 non-null    float64
 39  percblck  330 non-null    float64
 40  perchisp  330 non-null    float64
 41  blckpb    330 non-null    float64
 42  hispph    330 non-null    float64
 43  whtepw    330 non-null    float64
 44  blckph    330 non-null    float64
 45  hisppb    330 non-null    float64
 46  lsalary   353 non-null    float64
dtypes: float64(20), int64(27)
memory usage: 129.7 KB

There is a bunch of variables who describe characteristics of the 353 baseball players in the sample.

Subsetting¶

The process of accessing only a particular part of a dataframe is often called subsetting. There are, as usual several ways to do that.

Subsetting a single column/variable¶

Here we will see how to access particular columns in a dataframe. Say you wish to access the hits variable. Each of the following three commands do exactly the same, access the hits variable.

#df_bball['hits']
# df_bball.loc[:,'hits']    # use iloc if you are using a name
df_bball.iloc[:, 7]  # use iloc if using integers like col numbers to access

0      1939
1       863
2       840
3       816
4       928
       ... 
348     260
349     183
350     176
351     510
352       4
Name: hits, Length: 353, dtype: int64

Subsetting multiple columns/variables¶

If you want to access multiple columns you can do that in the following way. Say you want to access atbats to hits.

df_bball[['atbats', 'runs', 'hits' ]]
# df_bball.loc[:,'atbats':'hits']    
# df_bball.iloc[:, 5:8]  # 8 is the first column NOT included

Note that ['atbats', 'runs', 'hits' ] basically produces a list of variables. That list does not have to have the same order as the variables in the original dataframe.

Let's say you also want the salary variable. The easiest is to use a list.

sel_vars = ['salary', 'atbats', 'runs', 'hits' ]
df_bball[sel_vars]

Subsetting rows¶

If you want to subset a set of rows you can do this as follows if you wanted to view the 8th observation.

# df_bball.iloc[7,:]   # gives you the eights row/observation, recall we start counting at 0
df_bball.loc[7,:]   # gives you the eighth row/observation 

salary      2.750000e+05
teamsal     3.840738e+07
nl          1.000000e+00
years       1.000000e+01
games       4.320000e+02
atbats      1.005000e+03
runs        7.800000e+01
hits        2.400000e+02
doubles     3.500000e+01
triples     5.000000e+00
hruns       1.300000e+01
rbis        9.500000e+01
bavg        2.390000e+02
bb          3.900000e+01
so          1.400000e+02
sbases      1.000000e+00
fldperc     9.900000e+02
frstbase    0.000000e+00
scndbase    0.000000e+00
shrtstop    0.000000e+00
thrdbase    0.000000e+00
outfield    0.000000e+00
catcher     1.000000e+00
yrsallst    0.000000e+00
hispan      0.000000e+00
black       0.000000e+00
whitepop    5.772110e+06
blackpop    1.547725e+06
hisppop     8.934220e+05
pcinc       1.884000e+04
gamesyr     4.320000e+01
hrunsyr     1.300000e+00
atbatsyr    1.005000e+02
allstar     0.000000e+00
slugavg     3.223881e+01
rbisyr      9.500000e+00
sbasesyr    1.000000e-01
runsyr      7.800000e+00
percwhte    7.027797e+01
percblck    1.884423e+01
perchisp    1.087780e+01
blckpb      0.000000e+00
hispph      0.000000e+00
whtepw      7.027797e+01
blckph      0.000000e+00
hisppb      0.000000e+00
lsalary     1.252453e+01
Name: 7, dtype: float64

To get multiple rows you would call

#df_bball.iloc[5:7,:]   # gives you the first row/observation, recall we start counting at 0
df_bball.loc[5:7,:]
# df_bball[5:7]     # this also works if you are looking at multiple columns, but better to use .loc or .iloc

Objects and Pointers¶

Let's start with a little example. We define test which is defined as the first 5 observations of the salary variable.

test = df_bball.iloc[0:5,0]
test

0    6329213
1    3375000
2    3100000
3    2900000
4    1650000
Name: salary, dtype: int64

Now, for som inexplicable reason, you decide to change the first three values in test to 0.

test[0:3] = 0
test

0          0
1          0
2          0
3    2900000
4    1650000
Name: salary, dtype: int64

So, that worked. Let's have a look at the original dataframe from where we copied these data.

df_bball.iloc[0:5,0]

0          0
1          0
2          0
3    2900000
4    1650000
Name: salary, dtype: int64

What??!!! How did that happen? Here is what that original line of code test = df_bball.iloc[0:5,0] actually did. It did not create a "copy" in the usual way. What it actually did was to just create a new and alternative name for these particular five elements of df_bball.iloc[0:5,0]. Think of a street address and a PO.Box which delivers mail to the same street address. You could either use the street address or the PO.Box and your mail would go to the same place. This is what we did, test only points at df_bball.iloc[0:5,0], that is why it is also called a pointer.

So it is best to think of test not as a variable but rather as an object, one that points at another.

Clearly the above behaviour is not one we really want, if you thought that, once defined test is its own independent object. So what to do?

Return to the earlier line where you defined the new object test and replace it by test = df_bball.iloc[0:5,0].copy(). But before doing that we have to re-load the data.

df_bball = pd.read_csv(url, na_values = '.')
test = df_bball.iloc[0:5,0].copy()
test[0:3] = 0
df_bball.iloc[0:5,0]

0    6329213
1    3375000
2    3100000
3    2900000
4    1650000
Name: salary, dtype: int64

Observe the different behaviour. Now the values in the original dataframe df_bball have not changed.

By using the .copy() when you defined test you actually created what we would understand as being a copy. This you will have to keep in mind when you actually want to create a copy of data. Especially if you think data will be changed you need to carefully think about whether you will want them changed in the original dataframe or not.

Boolean variables and subsetting objects¶

Boolean variables are variables that take the value TRUE or FALSE. Let's illustrate this.

test = df_bball['salary'] > 1000000
test[0:7]

0     True
1     True
2     True
3     True
4     True
5    False
6    False
Name: salary, dtype: bool

For each of the 353 baseball players we asked the question whether they earned more than USD 1,000,000. We then looked at the first 7 observations in the sample and saw that the first 5 did indeed earn more than USD 1,000,000 (TRUE) but the next two did not (FALSE). In test you will find the answer to that question for all 353 baseball players.

You may be wondering how many in the sample did earn more than that threshold? sum(test) counts the number of TRUE values. np.mean(test) calculates the proportion and test.describe() tells you which answer was the most frequent.

sum(test)
# np.mean(test)
# test.describe()

143

You may wonder what the big deal is. In what comes in this walkthrough you will find some uses of Boolean variables. But here is one. Say you wanted to create a new dataframe with all the players that did earn more than USD 1,000,000, then here are a number of ways to do that (note that here we are creating copies).

millionaires1 = df_bball[test].copy() # where we use the variable defined above
millionaires2 = df_bball.loc[test].copy()
millionaires3 = df_bball.loc[test,:].copy()
millionaires4 = df_bball[df_bball['salary'] > 1000000].copy()
millionaires5 = df_bball.query('salary > 1000000').copy()

There is not a right or wrong way of doing this.

It is important to understand that you can create Boolean variables from a combination of conditions, say you wanted to select all players who earn more than USD 1,000,000 and have not more than 5 years of experience.

test2 = ( df_bball['salary'] > 1000000) & (df_bball['years'] < 6)
sum(test2)

28

There are only 28 players that fit this criterion. Here the two conditions were connected with the AND (&) opeator. You may also connect conditions with an OR (|) operator. There are a few more important operator you may need: IS EQUAL TO is represented by ==, UNEQUAL TO is !=.

if-else¶

An important tool in the toolbox of any applied economics and econometrician is the if-else condition. This is important as one will usually write code that works irrespective of certain conditions. Here we will demonstrate this with a simple example.

var1 = "salary"
var2 = "years"
# np.corrcoef delivers a 2x2 matrix, the corr is in the off-diagonal , eg [0,1]
c12 = np.corrcoef(df_bball[var1], df_bball[var2])[0,1]  

if c12 > 0:
    print("the correlation is positive")
else:
    print("the correlation is not positive")

the correlation is positive

You can also think of more than 2 cases.

if c12 > 0.1:
    print("the correlation is positive")
elif c12 < -0.1:
    print("the correlation is not positive")
else:
    print("the variables are basically uncorrelated")

the correlation is positive

Here, we only have a single line of code (print ( )), but there could be multiple lines of code after every condition. Also note that the code lines between if, elif and else are indent. This is important, the indent tells Python when the code ends and when the next condition comes.

Loops¶

Loops are a way to make your code repeat bits of code, potentially many, many times. Above we calculated a correlation between two variables. We have many variables (check df_bball.shape) to confirm that there are 47 variables altogether. We saw previously that they are all numerical variables. Let's say you wanted to figure out whether any of the variables is negatively correlated with salary. We would now have to calculate 46 correlations, the correlations of all variables, other than salary, with the variable salary. As salary is the first variable (at position 0), we need to loop through variables 1 to 46.

range(df_bball.shape[1])

range(0, 47)

words = ["Here", "I","want","to","concatenate","words","using","pipe","delimeter"]
print(" ".join([df_bball.columns[1], "is uncorrelated"]))

teamsal is uncorrelated

df_bball.columns[1]

'teamsal'

var1 = "salary"

for i in range(1,47):
    
    # np.corrcoef delivers a 2x2 matrix, the corr is in the off-diagonal , eg [0,1]
    c12 = np.corrcoef(df_bball[var1], df_bball.iloc[:,i])[0,1]  
    if c12 > 0.1:
        print(" ".join([df_bball.columns[i], "is positively correlated with salary"]))
    elif c12 < -0.1:
        print(" ".join(["!!!", df_bball.columns[i], "is negatively correlated with salary"]))
    else:
        print(" ".join([df_bball.columns[i], "is uncorrelated with salary"]))

teamsal is positively correlated with salary
nl is uncorrelated with salary
years is positively correlated with salary
games is positively correlated with salary
atbats is positively correlated with salary
runs is positively correlated with salary
hits is positively correlated with salary
doubles is positively correlated with salary
triples is positively correlated with salary
hruns is positively correlated with salary
rbis is positively correlated with salary
bavg is positively correlated with salary
bb is positively correlated with salary
so is positively correlated with salary
sbases is positively correlated with salary
fldperc is uncorrelated with salary
frstbase is uncorrelated with salary
scndbase is uncorrelated with salary
shrtstop is uncorrelated with salary
thrdbase is uncorrelated with salary
outfield is positively correlated with salary
!!! catcher is negatively correlated with salary
yrsallst is positively correlated with salary
hispan is uncorrelated with salary
black is positively correlated with salary
whitepop is uncorrelated with salary
blackpop is uncorrelated with salary
hisppop is uncorrelated with salary
pcinc is uncorrelated with salary
gamesyr is positively correlated with salary
hrunsyr is positively correlated with salary
atbatsyr is positively correlated with salary
allstar is positively correlated with salary
slugavg is positively correlated with salary
rbisyr is positively correlated with salary
sbasesyr is positively correlated with salary
runsyr is positively correlated with salary
percwhte is uncorrelated with salary
percblck is uncorrelated with salary
perchisp is uncorrelated with salary
blckpb is uncorrelated with salary
hispph is uncorrelated with salary
whtepw is uncorrelated with salary
blckph is uncorrelated with salary
hisppb is uncorrelated with salary
lsalary is positively correlated with salary

Only one variable is negatively correlated (as defined by a correlation smaller than -0.1). It turns out that very often loops are actually not necessary. The above could have been achieved by using the .corr() method applied to our dataframe df_bball.

df_bball.corr()

Looking down the first column (correlations with salary) you will find that only the correlation with the catcher variable is smaller than -0.1. Note that the catcher variable is a binary/dummy variable which takes the value 1 only for players which are catchers and 0 otherwise.

You can see that the .apply method can be used to apply functions to rows or columns.

Practice the loop¶

.tabset

The task¶

Calculate the sum of the numbers from 1 to 1000 using a loop.

The solution¶

k = 1000
sol = 0

for i in range(1,k+1):
    sol = sol + i
    
print(sol)

500500

¶

.unlisted|O_O|.unnumbered

The .apply method¶

With the apply method you can apply a function easily to each row or column of a DataFrame. It allows you to quickly perform operations on your data without needing to write complicated loops or manually iterate over each row or column. Let's illustrate this with an example. We will look at our dataframe df_bball. We wish to find the maximum across the ['salary', 'teamsalary', 'hits'] columns.

df_bball[['salary', 'teamsal', 'hits']].apply(max,axis=0) # applies the max functions to all columns (axis = 0)

salary      6329213
teamsal    42866000
hits           3025
dtype: int64

You could also find the max value across all variables for players 0 to 4.

df_bball.iloc[0:4,].apply(max,axis=1) # applies the max functions to all rows (axis = 1)

0    38407380.0
1    38407380.0
2    38407380.0
3    38407380.0
dtype: float64

For each player we actually pick out his team's salary and clearly they all play for the same team (with a salary of 38407380).

You can see that the .apply method can be used to apply functions to rows or columns.

Practice the .apply method¶

.tabset

The task¶

Calculate the mean of every variable in df_bball using the apply method. The mean can be calculated using the np.mean function.

The solution¶

df_bball.apply(np.mean,axis=0) # applies the max functions to all columns (axis = 0)

salary      1.345672e+06
teamsal     3.079483e+07
nl          4.759207e-01
years       6.325779e+00
games       6.484249e+02
atbats      2.168592e+03
runs        2.904023e+02
hits        5.847365e+02
doubles     1.038810e+02
triples     1.673371e+01
hruns       5.503116e+01
rbis        2.647450e+02
bavg        2.589858e+02
bb          2.120822e+02
so          3.404079e+02
sbases      6.580170e+01
fldperc     9.765099e+02
frstbase    1.274788e-01
scndbase    1.048159e-01
shrtstop    1.388102e-01
thrdbase    9.631728e-02
outfield    3.852691e-01
catcher     1.473088e-01
yrsallst    8.923513e-01
hispan      1.813031e-01
black       3.059490e-01
whitepop    3.984508e+06
blackpop    7.887030e+05
hisppop     9.006723e+05
pcinc       1.883314e+04
gamesyr     9.007604e+01
hrunsyr     7.119053e+00
atbatsyr    2.930280e+02
allstar     9.242893e+00
slugavg     3.953016e+01
rbisyr      3.505021e+01
sbasesyr    8.501081e+00
runsyr      3.858299e+01
percwhte    7.263109e+01
percblck    1.654893e+01
perchisp    1.081999e+01
blckpb      5.144886e+00
hispph      2.004179e+00
whtepw      3.666764e+01
blckph      2.931898e+00
hisppb      3.046493e+00
lsalary     1.349218e+01
dtype: float64

¶

.unlisted|O_O|.unnumbered

functions¶

Functions are at the core of all programming languages. In fact, even in this walk-through we have already used functions. Earlier we calculated the mean value of some variables. When you called the np.mean for instance, Python called up a function that calculated a mean of a variable. You could do that without you seeing the details of the mean calculation. That function was written by someone else and you could use that function as it was imported with the pandas package.

Here we will learn how to write a function ourselves. You would want to do that in case there is a combination of operations you may wish to repeatedly apply.

# here we define a function, sum_integers_to_k is the name of the function 
# it requires one input
def sum_integers_to_k(k):
  sol = 0

  for i in range(1,k+1):
    sol = sol + i
    
  return sol

Once you run this code, noting obvious will happen. However, Python will have stored this function in its memory and when you run the next line (sum_integers_to_k(40)), it will go to that code use 40 as the value of k, calculate the sum of all numbers to 40 and then return that value.

ans1 = sum_integers_to_k(40)
print(ans1)

820

Practice functions¶

.tabset

The task¶

Write a function that takes two numbers as inputs, k1 and k2, and sums up all the integers between these two numbers (including k1 and k2). The function should return that sum. In order to understand how to use two parameters you may have to find on the internet examples of functions that accept multiple inputs. You could use search terms like "Python functions with two parameters".

The solution¶

Adjust the code of the simple example to our problem at hand. See how this is done in the next step.

# here we define a function, sum_integers_to_k is the name of the function 
# it requires one input
def sum_integers_between(k1,k2):
    sol = 0

    for i in range(k1,k2+1):
        sol = sol + i
    
    return sol

And now you can call that function.

ans2 = sum_integers_between(10,40)
print(ans2)

775

¶

.unlisted|O_O|.unnumbered

lambda functions¶

Lambda functions are funny creatures and it may not be immediately obvious what their values are. Above we introduced functions as something you write when you want to repeatedly call the same set of code lines (with potentially different parameters). lambda functions are not that. lambda functions are used to do little things which you will only do once, i.e. only apply to one dataframe but not repeatedly to several dataframes. For instance we could consider writing a short function which checks whether a particular baseball player can be considered a veteran (very experienced player, years > 10). We use that to create a new variable Exper_cat.

temp = df_bball.copy()
temp['Exper_cat'] = temp.apply(lambda row: 'Veteran' if (row['years'] > 10)  else 'Not Veteran', axis = 1)

# Convert the 'Exper_cat' column to a categorical type
temp['Exper_cat'] = pd.Categorical(temp['Exper_cat'])

Here is what happened in the above line in which we define the new variable (temp['Exper_cat'] =). We use the .apply( .... , axis = 1) method meaning that we do something for every row of dataframe temp. Recall that in place of the four dots .... we would name the function we want to apply. However, Python does not have a is_the_player_a_veteran_function ready for use. We therefore need to write that function. And we write that function on teh spot (as we do not think that we will need it again later). So what did lambda row: 'Veteran' if (row['years'] > 10) else 'Not Veteran' do?

This started with lambda. This signals to Python that a special type of function is coming. This function has one input which here is labeled row but it could be labeled however you want. Here we call it row, as the apply method is applied by rows, so the apply method will "hand-over2 one row at a time to the lambda function. The rest of the line now determines what should happen. Here we are asking the function to return Veteran if the years variable is larger 10 and Not Veteran in case that is not correct.

This looks of course very much like an if else function, although its architecture is somewhat different (compare to the earlier section on if else functions).

The reason why this is incredibly useful is that the function can become a little more complicated allowing a new variable definition to depend on multiple other variables. Here we illustrate this by defining a new variable in the df_bball dataframe called position which indicates which position a player plays in. The definition depends on which of the positional dummy variables (frstbase to cathcher) is equal to 1.

df_bball['position'] = df_bball.apply(
    lambda row: 'First Base' if row['frstbase'] == 1 else 
                'Second Base' if row['scndbase'] == 1 else 
                'Shortstop' if row['shrtstop'] == 1 else 
                'Third Base' if row['thrdbase'] == 1 else 
                'Outfield' if row['outfield'] == 1 else 
                'Catcher' if row['catcher'] == 1 else None, axis=1
)

# Convert the 'position' column to a categorical type
df_bball['position'] = pd.Categorical(df_bball['position'])

Summary¶

In this walkthrough you have learned some basic programming techniques. You will encounter these whenever you try to understanding someone else's code or you have to develop code yourself that does more than just simple work.

This walk-through is part of the ECLR page.