Part 1 — Weather data: collection¶

At the time of my IB exams, I was in Hong Kong, so I navigated to the Hong Kong Observatory to download the csv files, and was very quickly reality-checked when I realized the data I sought were nowhere to be found. Eventually I found a subpage that had daily extracts of records for my timeframe, but the problem was, the page looked like this:

And there was no 'Download' button in sight. In defeat, I copy-pasted 2019's records for March, April, May, and June (although June was not in the Toggl data) into a Jupyter notebook and prepared for some wrangling. With each month imported as a string, by some miracle, the text was delimited with tabs:

We ideally want this in a pandas data frame or in .csv format, so we can begin by stripping any newlines and replacing the delimiter.

Then, we can aggregate all months into one yearly string, and format the dates properly by joining 2019/0 with the month and date:

The data also have some special characters indicating partial records, but we'll just strip those too, and add our headers. This gives the following:

Now we can write this to a file and load it with pandas.

Our last preprocessing job is to tell pandas our date column is of datetime format, and our job is done.

Because I'm curious, let's check out how temperature and rain are distributed over the four months:

This is not just an excuse to show off the sheer power of Altair (the chart is interactive, too), but we can see that there are not that many rainy days in our dataset. Either way, we've essentially got half of the data processing done.

Part 2 — Toggl data: quantifying productivity¶

I glossed over this in the introduction, but if we're going to approach the question "Does weather impact my productivity?" we're going to need to define 'productivity.' Before we tackle that, let's load the data from Toggl:

Understanding the Toggl data¶

In Toggl, each entry requires a 'description' and a 'project' (originally designed for firms to track employee hours). Here, a 'description' in the data represents a specific task done, e.g. Physics Notes, and a 'project' represents what type of task that belongs to, e.g. Study / Revision. Toggl also records the date and time you started and ended the task, in addition to the duration.

The data frame is obviously quite bloated (and includes my email!), so let's subset the columns:

Defining productivity¶

So how can we quantify productivity? There would be many ways to do this, but my approach is simple: assume that productivity is directly proportional to the minutes of schoolwork logged (in some timeframe). This way, if I goofed off, I wouldn't have recorded any work, so Minutes would be 0. An extra benefit of totalling the minutes worked rather than the amount of tasks done in a day is that we don't interpret doing many small tasks as 'being more productive' than doing one large task (also, we lack sufficient data to count entries as units of productivity).

It may be argued this approach is counterintuitive given productivity is largely thought of alongside efficiency, and this approach may actually penalize efficiency by rewarding longer tasks (even if the effort is low). However, since our sample is quite small, my intuition says we can't reliably derive a measure like minutes worked per complete task, so we'll stick to the prior approach. As an aside, this is an example of an assumption and design decision made in the data science process that may fundamentally alter the results of an analysis.

With that out of the way, let's add a new row to our data frame called Minutes that is the total minutes worked for one entry. We can pass the string Duration to a function to convert the HH:MM:SS format to a numeric type:

The map() function here is essentially taking every record of Duration and passing it to str_to_dt() to compute the minutes worked. Let's take this opportunity to also correct some types, and we'll add a new column called Task_Start to store both the date and time of starting a time entry.

Now we're ready to visualize the data. The two plots effortlessly made with Altair display how minutes are distributed in each project, and when each task was done. Click on a colored bar in the top plot to explore a project, and hover over points on the lower chart to see individual tasks. You can also try clicking and dragging on the lower plot to highlight a timeframe to see the minutes distribution update.

Unsurprisingly, homework and revision dominate the entries. The 'Entertainment' category is also quick to fade as exams begin to creep up (beginning on May 6 in 2019). There are also some hints my study habits were more sporadic towards the beginning of exams too, with entries becoming more densely distributed at shorter minute counts.

We can also inquire into the day specifics. The two charts below implicitly answer the questions "Which day was I most productive throughout the week?" and "At what time did I record the most tasks?"

These charts show entertainment and homework taking over during weekends, and revision stagnating throughout the week (remember, I was on study-leave for these months).

Part 3 — Answering the question(s)¶

Let's move on to trying to answer the question at hand: "Does the weather affect my productivity?"

Rain analysis¶

First, I want to investigate whether rain makes me more prone to working hard. We're aiming for data with just productive entries, so let's drop all rows that aren't concerning 'Homework' or 'Study / Revision':

Now let's take a look at our minutes data for 'productive' entries:

As an aside, can we investigate which subject/task took me the longest to complete work for, on average:

So English seems to be the most laborious with Mathematics past papers trailing. Economics also has the highest entry total of 72. I could aggregate these further into classes, but I digress.

At this point we should ask about the number of days we have:

39 is a pretty small sample size, but the analysis is still worth a shot anyways. Let's start by just grabbing the related rows of our weather data frame:

Then we can aggregate our Toggl data into two rows: days, and minutes worked for that day.

This operation isn't too intimidating — in English, this is telling pandas to compute the sum of minutes worked for each day, and .reset_index() fixes our indices (left-most bolded column), and we finish by renaming the columns.

A nice result. We can now seamlessly join these tables and write them to a .csv:

To gauge whether rain has an effect on minutes worked, we want to be able to classify each day as either 'rainy' or 'not rainy'. One method for classifying days based on precipitation is to use a threshold $\boldsymbol{\lambda} \in \{0.1,0.2,1,10,25\}$ such that:

To choose $\lambda$, let's check out our data for rain:

Given Hong Kong's humid nature alongside the median precipitation, $\lambda=1\,\text{mm}$ appears to be a good fit for the data. We can redefine our rain column to achieve the following data frame:

Let's see how the data are split:

Which is as good of a split as we could've hoped for. Now for some preliminary visualization: let's see how rainy/non-rainy days are distributed against minutes:

By eye, it looks like rainy days have slightly higher minute counts. Let's compute the mean of both types of days:

So the average task time is higher on average, but is the difference statistically significant? One way of answering this is to knock on classical statistics' door and ask for a p-value. We want to test the alternative hypothesis that rainy days, on average, have higher minute counts. So our null hypothesis will be that there is no difference between minutes worked on average on rainy and non-rainy days. So our set-up is as follows:

where $\mu_R,\mu_N$ are the true means of minutes worked for rainy and non-rainy days respectively. Let's write the data frame to a .csv and head into R to perform a simple t-test:

rain <- read.csv("rain_school.csv")
t.test(rain[rain$rain=="Yes",]$minutes, rain[rain$rain=="No",]$minutes, alternative='greater')

    Welch Two Sample t-test

data:  rain[rain$rain == "Yes", ]$minutes and rain[rain$rain == "No", ]$minutes
t = 0.48793, df = 35.485, p-value = 0.3143
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 -40.11065       Inf
sample estimates:
mean of x mean of y 
 216.9430  200.6475 

So we obtain a p-value of $\approx0.3$, which is quite large. To interpret this, we can say that if we repeated the experiment for minutes worked on rainy/non-rainy days, and there really was no impact of rain on minutes worked, the probability of seeing more minutes worked for rainy days than non-rainy days to the extent we saw or to a greater extent just by chance is 30%.

While that is a mouthful, it tells us there is a relatively high probability of seeing a positive deviation in minutes worked just by chance if there was no difference in rainy/non-rainy days — so we fail to reject the null hypothesis. The mini-conclusion here is that more data are definitely needed.

Predicting minutes worked¶

Let's backtrack and look at the rest of the weather data:

There are an abundance of variables — let's just choose mean_temp, humid, cloud, rain, and sunshine as factors we think may have an impact on productivity.

Let's join this with our previous table containing the minutes, and write that to a .csv:

If you were wondering, sunshine is the number of hours of bright sunshine hours, not total. Let's head over to R to try to build some models for predicting minutes. We can begin by just checking out the correlations between our variables:

> days <- read.csv("full_days.csv")
> cor(days[c(2:7)])
            minutes   mean_temp      humid       cloud        rain    sunshine
minutes    1.00000000  0.08525068  0.1479811 -0.07462029 -0.04984429  0.07714833
mean_temp  0.08525068  1.00000000 -0.3882086 -0.63293077 -0.23396924  0.70265663
humid      0.14798108 -0.38820860  1.0000000  0.56440892  0.53635898 -0.63297290
cloud     -0.07462029 -0.63293077  0.5644089  1.00000000  0.33323933 -0.90797842
rain      -0.04984429 -0.23396924  0.5363590  0.33323933  1.00000000 -0.38933099
sunshine   0.07714833  0.70265663 -0.6329729 -0.90797842 -0.38933099  1.00000000

We can see the weather associations we'd expect — temperature/cloud and sunshine, humidity and cloud, etc. But the correlations among minutes look pretty unpromising. Regardless, let's begin by partitioning our data with a 75/25 training/testing split:

set.seed(30734592) # for reproduceability
train.ind <- sample(seq_len(nrow(days)), size = floor(.75*nrow(days)))
days.train <- days[train.ind,]
days.test <- days[-train.ind,]

And fit a multiple regression equation by modeling minutes as a function of our weather variables:

> days.fit <- lm(minutes ~ mean_temp + humid + cloud + rain + sunshine, data=days.train)
> summary(days.fit)

Residuals:
    Min       1Q   Median       3Q      Max 
-117.588  -65.600   -2.683   69.162  151.200 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -139.4934   413.7243  -0.337    0.739
mean_temp    -13.0997     9.7955  -1.337    0.194
humid          7.0194     4.1107   1.708    0.101
cloud          0.2519     2.0980   0.120    0.905
rain          -1.7436     1.2900  -1.352    0.190
sunshine      10.9424    14.1435   0.774    0.447

Residual standard error: 86.59 on 23 degrees of freedom
Multiple R-squared:  0.1723,    Adjusted R-squared:  -0.007695 
F-statistic: 0.9572 on 5 and 23 DF,  p-value: 0.4639

Our p-values are pretty large across the board, but there are some hints a few variables may be associated with minutes. We can calculate the Root Mean Square Error (RMSE) by

where $ y_i $ are the true values for our targets, and $\hat{y}_i$ are the predicted values, and $n$ is our sample size. This can be done in R for our test dataset:

> sqrt(mean((days.test$minutes - predict(days.fit, days.test))^2))
[1] 160.763

Which is quite a large average difference. We can try a few other models — bagged regression trees are generally good for reducing variance. We can create one in R with the rpart package:

rain_bag <- bagging(
    formula = minutes ~ mean_temp + humid + cloud + rain + sunshine,
    data = days.train,
    nbagg = 100,
    coob = TRUE,
    control = rpart.control(minsplit=2, cp=0)
)

> sqrt(mean((days.test$minutes - predict(rain_bag, days.test))^2))
[1] 154.6983

And we achieve a new RMSE of 155, which is a slight improvement over 161. We can try one more model — a random forest, for fun. We can use the randomForest library in R to elegantly do this:

rain.rf <- randomForest(minutes ~ mean_temp + humid + cloud + rain + sunshine,
                        data=days.train,
                        importance=TRUE)

> sqrt(mean((days.test$minutes - predict(rain.rf, days.test))^2))
[1] 149.7549

And an RMSE of 150 is again an improvement over 155 and 161, but is still very high. To conclude, we can plot the predictions and compare models:

Conclusion¶

As is typical with analysis of small samples, more data are needed to conclusively suggest there is any difference in (my) productivity owing to the weather. Looking back at the linear model, perhaps an interesting route to explore is humidity due to its notable t-statistic and seemingly borderline association ( $p\approx0.1$). Of course, this may very well be solely due to chance (the more low-quality features we include, the more likely we find a non-existent correlation due to chance).

At the very least, I'm happy I had the opportunity to poke around in some of my old data and refresh my pandas/R/Altair skils, and conduct some surface-level exploratory analysis even if the results were inconclusive. As is sometimes the case with throwaway personal projects, the famous saying can be invoked — it's about the journey, not the destination.