Forecasts that keep score

Problem

I work at a K-12 online learning provider: schools in the US and abroad partner with us so their students can take courses online. When students hand in assignments, a team of staff grades them, and we aim to grade each one within a set time. Handed-in work waits in a shared list, a “queue”, until someone grades it.

Weekends are the hardest stretch to plan: students keep handing in work, but fewer people are grading. The team’s leads wanted to know on Friday how much would still be waiting on Monday morning, so they could plan who grades what.

Constraints

• I had only one count of the queue per day, taken at slightly different times.
• How much the team grades over a weekend changes from week to week.
• The prediction had to be ready before Friday ended, and honest about how sure it was.

What I built

A Friday forecast, written in Python. It starts from one simple fact: anything waiting on Friday that is not graded over the weekend will still be waiting on Monday. The hard part is predicting how much the weekend clears.

Each forecast gives a range rather than one number, lists its assumptions, and states in advance which Monday result would prove it wrong. My first forecast broke two of my own rules: I typed its numbers in by hand, and I narrowed its range, which made it look more certain than it was. After that, every number came straight from the saved daily counts, every range was widened rather than narrowed, and every forecast was a script. On Monday, the check first re-runs Friday’s script to confirm it gives exactly the same answer, and only then scores it.

Before the forecasts, I did the analysis behind a one-week push, co-led with a manager on my team, to bring the weekend workload down.

How I proved it worked, and where it did not

Every Monday I took the first morning count and scored Friday’s forecast against its own rule. The scorecard keeps every miss.

• First idea: the weekend clears a fixed amount. The first forecast assumed the team grades about the same amount every weekend. I already knew this was shaky: two past weekends fit three different explanations equally well, and those explanations disagreed by a factor of three. Monday came in far above the range, which ruled the idea out.
• Second idea: the weekend clears a fixed share. I switched to assuming the weekend clears a steady share of whatever is waiting on Friday. That held twice, missed once just below its range, and missed once well above it, right after the biggest Friday total since I adopted it.
• A miss I had warned about in advance. That weekend, the amount graded stayed flat while Friday’s total grew by more than half, so more was still waiting on Monday. The forecast’s written rule had said this would happen if weekend grading does not grow with the workload.

Outcome

• Two of five forecasts landed inside their range. The record is shown as it is.
• Each miss improved the method: the fixed-amount idea was dropped first, then the fixed-share rule was marked unsafe after a big Friday.
• Every forecast can be re-run from its saved numbers, so any disagreement is about the method, never about which numbers were used.

What I would do next

• Forecast each subject team separately: spare time on the math team cannot grade a science assignment.
• Publish a chart showing how often each forecast range held.

The demo it pairs with

Workload forecaster. Set how many people grade over the weekend and how much new work arrives, pick “fixed amount” or “fixed share”, then reveal Monday and see the forecast scored. A scorecard of past weekends shows the hits and the misses.