Sorting courses by their neighbours

Problem

I work at a K-12 online learning provider: schools in the US and abroad partner with us so their students can take courses online. Student assignments are graded by teams of subject specialists, such as math, science or world languages, so every new course has to be handed to the right team. A daily check suggested a team based on words in the course title, and a person made the final call. The word rules had gaps: some subjects had no rule at all, and about 1 suggestion in 6 came out as “unclear”.

Constraints

• I could only read the course data. A suggestion is advice for a person, never an automatic change.
• Course titles are short, inconsistent and sometimes misleading. A place name, for example, can look like the name of a language.
• It had to fit into the existing daily check without changing how people work.

What I built

A recommender written in Python. Rather than reading the title alone, it looks for evidence about the new course, strongest first:

1. Siblings: the same course with a different part or term label, such as “A” and “B”.
2. Shared chapters: existing courses whose chapter titles overlap. Each similar course gets a vote, weighted by how much it has in common.
3. Rarer title words, weighted so that very common words count for little.
4. A word rule, only as a last resort.

Each suggestion comes with a confidence level (High, Medium or Low) and the similar courses behind it, so the person deciding can see why. I then rewrote the program in JavaScript to run inside the daily check, and confirmed that both versions give the same answer and confidence on every course. The daily check uses a lighter version without the chapter step, which I tested on its own; it also passed. It shows two similar courses beside each suggestion.

How I proved it worked

Before running any test, I wrote down the pass mark: High-confidence suggestions had to be right at least 95% of the time, and beat the old word rules on the same courses.

Then I hid each existing course’s real team, one at a time, and asked the program to work it out from everything else. Sibling courses are often set up together by the same person, which could make the results look better than they are, so I also ran a stricter test that hid every course created on the same day. Any adjustment after the pass mark was set was logged with its time, and the pass mark itself never moved.

Outcome

• Right about 99 times in 100, compared with about 93 for the old word rules on the same courses.
• Left only about 1 course in 70 without a suggestion, compared with about 1 in 6 before.
• The stricter same-day test still passed.
• Half of the worst misses were not the program’s fault: the recorded team looked wrong, and those went to a person to check. The other half were families of courses split between two teams, where the right answer depends on a rule nobody had written down yet.
• It has run in the daily check since October 2026.

What I would do next

• Add the chapter step to the daily check, and retire the word rules once the test below passes.
• Run the real-world test I committed to in advance: record suggestions for the next batch of new courses before anyone assigns them, then compare them with the real decisions a week later. If High-confidence suggestions are right less than 95% of the time, the method fails.

The demo it pairs with

Course matcher. Type a course title and a few chapter titles, or pick an example. See the most similar courses, their votes, the suggestion and its confidence, next to what the old word rule would have said.