GitHub Faces Performance Hurdles in August 2025
James Ding Sep 11, 2025 15:31
In August 2025, GitHub experienced three significant incidents affecting service performance, prompting upgrades to monitoring and safeguards to prevent future disruptions.
In August 2025, GitHub encountered three critical incidents that led to degraded performance across its services, as reported by GitHub. These incidents highlighted areas for improvement in the platform's infrastructure and monitoring systems.
August 5 Incident
The first incident occurred on August 5, lasting 32 minutes, due to a production database migration error. The migration aimed to drop an unused column from a table supporting pull request functionality. However, the Object-Relational Mapping (ORM) system still referenced this column, resulting in elevated error rates affecting pushes, webhooks, notifications, and pull requests. Approximately 4% of web and REST API traffic was impacted. The issue was initially mitigated by instructing the ORM to ignore the removed column, but a secondary incident affected around 0.1% of pull request traffic. GitHub plans to enhance automation and safeguards to prevent similar issues in the future.
August 12 Incident
On August 12, a more prolonged outage occurred when GitHub's search functionality was degraded for over three hours. Users experienced issues such as inaccurate search results and failures to load certain pages. The incident was traced back to connectivity problems between load balancers and search hosts, which were exacerbated by retry queues overwhelming the load balancers. This led to a peak failure of 75% of search queries. The issue was resolved by throttling the search indexing pipeline and rebooting a search host, which restored connectivity.
August 27 Incident
The final incident on August 27, lasting 46 minutes, saw Copilot and other services experience degraded performance. Similar to the first incident, it was caused by a database migration error where a column drop led to 5xx responses. GitHub has since blocked all drop column operations as a temporary measure and is working on implementing graceful degradation to ensure Copilot issues do not impact other services.
In response to these incidents, GitHub is enhancing its monitoring capabilities and refining its internal processes to prevent future disruptions. Users can follow real-time status updates on the GitHub status page.
Image source: Shutterstock.jpg)