Online courses housed in a secure LMS can have hundreds of hyperlinks on HTML pages that point to public websites as means of additional content or supplemental resources for students. Validating those hyperlinks across hundreds of courses, several times year, is a very manual time-consuming process. Hyperlinks break often and this is becoming a growing problem as the number of open learning resources available publicly on the web grows.
To solve this problem, I developed software to automate the process of checking links using C# .Net Core, Azure Durable Functions, Azure SQL, Azure KeyVault, and Angular. My application relies on the Brightspace API to securely access HTML pages. This application has a lot of parts working together built with separation of concerns in mind for performance, scalability, and user-experience.
The proof of concept was initially developed as a monolith and I quickly realized that this was going to be an issue for performance. The nature of validating hyperlinks is not a quick process as it relies on getting a response back from each website’s server and the speed of those servers is out of my control.
I used Azure Durable Functions for long-running services. The application scales out and links are validated in parallel for performance and user-experience. The information about the hyperlink, status of the web page, and location of the hyperlink in the course are stored in a SQL database. The scraping function is rate-limited to respect the Brightsapce server bandwidth. Link validation is only rate-limited when checking more than one web page on the same domain to prevent access issues and respect the site owners server limits.
Reports are displayed using Angular CLI and Bootstrap. When a new report is generated, the results are updated with the current status to keep the user informed of the progress. The ability to run new reports is rate limited to respect API bandwidth limitations. The Demo is connected to a live instance of the Brightspace LMS. Click "Run a new Report" to see it in action.
The Brightspace API does not have an SDK for .Net Core which meant that I wrote my own access method and hashed tokens. I developed a Brightspace API gateway separately, using KeyVault to secure access key pairs. This allows me to use the same code for future applications. I can use a different access key pair for each application.
The Brightspace API does not have a method for opening internal hyperlinks. Checking internal hyperlinks that go to an authenticated page required that I use HTTP client to manually log into the secured LMS. This required generating headers, cookies, and tokens in a C# Azure function.
Furthermore, I developed a “widget” that displays the results of the latest report, directly in each course.
Instructors would be prompted to fix any links as soon as they log into the online course in Brightspace. The widget was created with Angular and deployed to AWS.
While there is a delay with the consumption tier of Azure functions, an option for better performance would be warm starts. Furthermore, using Angular to keep the user notified of the current status allows for a better user-experience. Backing that with a database means that the user doesn’t even have to stay on that screen.
There are several possibilities for implementation. This could be configured to run nightly and check hundreds of courses with email reports of broken links.