Header Image
Active Incident

Updated 2 hours ago

osuosl.org HTTPS certificate not trusted on Amazon Linux and some Java runtimesDegraded Performance

Incident Status

Degraded Performance

Components

Core IPv6 Network, Core IPv4 Network, DNS, Mail Relays, FTP Mirrors, Internal Monitoring

Locations

Chicago, IL, Madison, WI, New York, NY, Salem, OR


November 4, 2026 11:20AM PST
November 4, 2026 7:20PM UTC
IDENTIFIED

Some clients can no longer verify the HTTPS certificate on most of our osuosl.org sites, including the ftp.osuosl.org mirrors (ftp-nyc, ftp-chi and ftp-osl). Downloads fail with "unable to get local issuer certificate" (curl error 60) or a Java "unable to find valid certification path" error. Who is affected: - Amazon Linux 2023 and Amazon Linux 2 hosts with the latest ca-certificates package (ALAS2023-2026-3306, ALAS2-2026-4037), which removed every eMudhra root. - Java runtimes whose trust store has dropped eMudhra's emSign Root CA - G1. - Anything else that has removed all eMudhra roots, such as a corporate trust policy. Not affected: Web browsers and most Linux distributions still trust this certificate, and the sites themselves are up. What happened: The *.osuosl.org certificate most of our sites use comes from InCommon and chains to eMudhra's emSign Root CA - G1. After last month's eMudhra key compromise, which involved a different eMudhra hierarchy, some vendors removed all eMudhra roots, including this one. The Jenkins project has temporarily taken our mirrors out of its download rotation for this reason. What we're doing: We are moving the ftp.osuosl.org mirrors to Let's Encrypt certificates, which we expect to finish in the next couple of weeks, and will post here when they have switched. Our other osuosl.org certificates will move off eMudhra before the current certificate expires in February. If you are affected and need help in the meantime, email support@osuosl.org.
Critical Linux kernel TCP vulnerability (CVE-2026-98432)Operational

Incident Status

Operational

Components

Internal Monitoring

Locations

Salem, OR


November 9, 2026 2:07PM PST
November 9, 2026 10:07PM UTC
IDENTIFIED

Update (Nov 9): Thursday's reboots of our CI runners and OSL services move one hour later, to Thursday, November 12, 12:00 PM – 4:00 PM PST (20:00 – 00:00 UTC). Friday's window, November 13, 10:00 AM – 1:00 PM PST (18:00 – 21:00 UTC), is unchanged. Thursday: https://status.osuosl.org/pages/maintenance/526712bf10cd4e587b000014/6af2441f50ddf6040ebabce9

November 4, 2026 1:02PM PST
November 4, 2026 9:02PM UTC
IDENTIFIED

Update (Nov 4): Our own internet-facing servers move onto fixed kernels in two maintenance windows. On Thursday, November 12, 11:00 AM – 3:00 PM PST (19:00 – 23:00 UTC), we reboot our CI runners and OSL services. On Friday, November 13, 10:00 AM – 1:00 PM PST (18:00 – 21:00 UTC), we reboot the project sites, mailing lists and FTP mirrors we host. The firewall rule on those servers stays in place until then. Thursday: https://status.osuosl.org/pages/maintenance/526712bf10cd4e587b000014/6aeb9bd7f7bb4b038d276f4d Friday: https://status.osuosl.org/pages/maintenance/526712bf10cd4e587b000014/6aeb9bf5b435fe0422769f92 These reboots update our servers only. If your VM runs AlmaLinux, run "dnf update kernel" and reboot into the new kernel, as in our Nov 2 update. If you need help, email support@osuosl.org.

November 2, 2026 1:47PM PST
November 2, 2026 9:47PM UTC
IDENTIFIED

Update (Nov 2): AlmaLinux's fixed kernels for CVE-2026-98432 are now in the regular updates for AlmaLinux 8 (4.18.0-553.182.1), 9 (5.14.0-687.64.1) and 10 (6.12.0-211.72.1). If your VM runs AlmaLinux, run "dnf update kernel" and reboot into the new kernel; you can then remove the interim nftables rule. For other distributions, check your vendor's security notices. On our side, the OpenStack servers move onto fixed kernels on Tuesday, November 3 (live migrations, no restart) and Thursday, November 5 (VMs restarted once), in the two maintenance notices on this page. Those reboots update our servers only, so VMs still need their own kernel update. If you need help, email support@osuosl.org.

November 1, 2026 10:20AM PST
November 1, 2026 6:20PM UTC
IDENTIFIED

A critical flaw in the Linux kernel's IPv4 TCP code (CVE-2026-98432, CVSS 9.8) lets a remote attacker with no login crash the kernel, and possibly run code, by sending crafted packets to any open TCP port. If your project runs Linux on an OpenStack VM or a server that you manage yourself, please install your distribution's fixed kernel as soon as it is available and reboot into it. Who needs to act: - Projects that manage their own OpenStack VMs or colocated servers running Linux. Every distribution is affected. - Hosts that OSL manages for you: we have already applied the interim mitigation below and will reboot them onto fixed kernels in our normal maintenance windows, announced here as usual. Fix status (Nov 1): - Debian: fixed. Debian 13 in 6.12.113-1 (DSA-6568-1), Debian 12 in 6.1.190-1 (DLA-4841-1). - AlmaLinux 8, 9 and 10: fixed kernels are in the testing repository (4.18.0-553.180.2, 5.14.0-687.61.2, 6.12.0-211.69.2) and not yet in the regular updates. - RHEL and Ubuntu: no fixed kernel yet; Red Hat rates the flaw Critical and Ubuntu has fixes in progress. For other distributions, including those built from RHEL, check your vendor's security notices. Interim mitigation: Until you can reboot onto a fixed kernel, drop incoming IPv4 TCP packets that carry IP options, which normal traffic does not use. Red Hat's bulletin gives this nftables rule (run as root; it lasts until the next reboot): nft add table inet cve_2026_98432 nft add chain inet cve_2026_98432 input '{ type filter hook input priority -310 ; policy accept ; }' nft add rule inet cve_2026_98432 input ip protocol tcp ip hdrlength gt 5 counter drop If the VM runs Docker or other containers, add the same chain and rule with "forward" in place of "input" so container ports are covered too. Remove it after rebooting onto the fixed kernel with: nft delete table inet cve_2026_98432 Red Hat's bulletin: https://access.redhat.com/security/vulnerabilities/RHSB-2026-016 If you are unsure whether you are affected or need help, email support@osuosl.org.

Scheduled Maintenance

Kernel Security Reboots for CVE-2026-98432 – Thursday, Nov 12, 12:00–4:00 PM PST (20:00–00:00 UTC)Planned Maintenance

Schedule

November 12, 2026 12:00PM - 4:00PM PST
November 12, 2026 8:00PM - 12:00AM UTC

Components

Internal Monitoring

Locations

Salem, OR

Description

Time change: Thursday's reboots now run 12:00 PM – 4:00 PM PST, one hour later than first announced. This replaces the earlier notice. On Thursday we will reboot our CI runners and some of OSL's own services to fix a new Linux kernel flaw, CVE-2026-98432. This is separate from the October 30 reboots for Fragnesia (CVE-2026-46300). It is a different flaw, and its fix was released on November 2, after those reboots. Each server reboots once during the window. Date: Thursday, November 12, 2026 Time: 12:00 PM – 4:00 PM PST (20:00 – 00:00 UTC) Expected impact: each CI runner stops taking jobs for about 20 minutes; each other service is down for about 5 minutes What's happening: The update fixes CVE-2026-98432, the remote flaw in the Linux kernel's TCP code described in our earlier notice on this page. The kernels we installed for Fragnesia don't include this fix. A firewall rule has blocked it on these servers since November 1, and the fixed kernels are already installed, so a reboot is the only step left. Affected: - Our GitLab CI runners for x86_64, aarch64, ppc64le and s390x, one runner per architecture at a time. Each stops taking new jobs and gives running jobs up to 10 minutes to finish before it reboots; a job still running then is stopped and needs a retry. s390x has a single runner, so s390x jobs wait in the queue for about 20 minutes. - PowerCI and IBM Z CI Jenkins: builds running when their server reboots need to be restarted. - git.osuosl.org, jenkins.osuosl.org and supermarket.osuosl.org - OSL's chat server (chat.osuosl.org) - Our support ticket system (support.osuosl.org). Email to support@osuosl.org sent during the reboot is held and delivered afterwards. - OSL shell accounts, and a few other OSL VMs - Sites behind our load balancers, including osuosl.org and phpBB, may see a few seconds of dropped connections twice, as each load balancer hands over to its partner and back. Not affected: - OSL's mail relays and DNS servers are rebooted one at a time with their partners still serving, so mail and name service stay up. - Project sites, mailing lists and our FTP mirrors reboot the next day, Friday, November 13 (separate notice). If a service doesn't come back after its reboot, or you notice any issues after the work is complete, please email support@osuosl.org to open a ticket so we can resolve them. Questions or concerns before then can go to the same address. We apologize for the interruption.
Kernel Security Reboots for CVE-2026-98432 – Friday, Nov 13, 10:00 AM–1:00 PM PST (18:00–21:00 UTC)Planned Maintenance

Schedule

November 13, 2026 10:00AM - 1:00PM PST
November 13, 2026 6:00PM - 9:00PM UTC

Components

FTP Mirrors, Internal Monitoring

Locations

Chicago, IL, New York, NY, Salem, OR

Description

On Friday we will reboot the servers behind the OSL-hosted services listed below to fix a new Linux kernel flaw, CVE-2026-98432. Many of these services were also rebooted on October 30 for Fragnesia (CVE-2026-46300). This is a different flaw, and its fix was released on November 2, after those reboots, so they need one more. Each service will be down for a few minutes, once, during the window. Date: Friday, November 13, 2026 Time: 10:00 AM – 1:00 PM PST (18:00 – 21:00 UTC) Expected impact: about 5 minutes of downtime per service while its server reboots; the two GitLab instances can take longer to come back What's happening: The update fixes CVE-2026-98432, the critical remote flaw in the Linux kernel's TCP code described in our earlier notice on this page. The kernels we installed for Fragnesia don't include this fix. A firewall rule has blocked it on these servers since November 1, and the fixed kernels are already installed, so a reboot is the only step left. Affected: - phpBB websites and services - Linux Foundation sites we host, including their mailing lists, archives, bug tracker and FOSSology - Mailing lists on lists.osuosl.org - OSRF / ROS websites, including the ROS wiki, docs and answers sites - Snowdrift.coop, including CiviCRM and its mail - OpenPOWER Foundation websites and services - Nura's GitLab and U-Boot's GitLab - Deluge, OpenMRS, Mozillazine, Apereo and OGRE websites - Buildroot's source mirror - Inkscape's Matrix server, GFortran's Mattermost and SeaGL's Nextcloud - Cinc project services - Websites on our shared web hosting, WebDAV, and projects hosted for CASS Software Development Group (SDG) - OpenID Foundation, Replicant, Mulgara and StreamWebs sites, and our Etherpad and container registry - Our FTP mirrors, one at a time, so ftp.osuosl.org keeps serving throughout Not affected: - OSL's mail relays, DNS servers and load balancers are rebooted one at a time with their partners still serving, so mail, name service and web traffic stay up. - VMs that projects manage themselves are not restarted. Please update and reboot those yourselves (see our earlier notice). - Our CI runners and some OSL services reboot the day before, on Thursday, November 12 (separate notice). You don't need to do anything. If a service doesn't come back after its reboot, or you notice any issues after the work is complete, please email support@osuosl.org to open a ticket so we can resolve them. Questions or concerns before then can go to the same address. We apologize for the interruption.
FTP Protocol Retirement – Tuesday, Dec 15, 9:00–10:00 AM PST (17:00–18:00 UTC)Planned Maintenance

Schedule

December 15, 2026 9:00AM - 10:00AM PST
December 15, 2026 5:00PM - 6:00PM UTC

Components

FTP Mirrors

Locations

Chicago, IL, New York, NY, Salem, OR

Description

On Tuesday, December 15, 2026, we will permanently turn off FTP (ftp://) access to ftp.osuosl.org and the other OSU Open Source Lab mirror hostnames. The files themselves are not going anywhere: everything stays available over HTTP and HTTPS, and most of it over rsync. Date: Tuesday, December 15, 2026 Time: 9:00 AM – 10:00 AM PST (17:00 – 18:00 UTC) Expected impact: from this time on, FTP connections to the mirrors will be refused. HTTP, HTTPS and rsync keep working throughout. What you need to do: If you or your users download from our mirrors with an ftp:// URL, switch it to HTTPS before December 15. The paths are the same, so ftp://ftp.osuosl.org/pub/<project>/ becomes https://ftp.osuosl.org/pub/<project>/. If you reach us through a project name on another domain, such as ftp.<project>.org, use https://ftp.osuosl.org/ for HTTPS for now: those names don't have their own certificates yet, and we're working on adding them. If you run a mirror that syncs from us over FTP, switch to rsync://rsync.osuosl.org/<module>/ instead. Not every project we mirror has an rsync module; if the one you need is missing, email support@osuosl.org to request it. Affected: FTP on ftp.osuosl.org, ftp-osl.osuosl.org, ftp-chi.osuosl.org and ftp-nyc.osuosl.org, and on any other name that points at our mirrors, through a CNAME to ftp.osuosl.org or our IP addresses (for example ftp.<project>.org or <project>.osuosl.org). Not affected: HTTP and HTTPS at https://ftp.osuosl.org/, rsync at rsync://rsync.osuosl.org/, and every hostname above over HTTP and rsync (HTTPS on ftp.osuosl.org and <project>.osuosl.org names). The names stay; only the FTP protocol goes away. Why: FTP is an old protocol that most major projects have moved away from. Anonymous FTP has no practical way to encrypt a download or confirm that a file wasn't altered on the way to you; HTTPS does both, which is why we point you there. FTP also opens a separate connection on a negotiated port for every transfer, which makes it hard to run reliably behind today's firewalls, NAT and load balancers, and it can't be cached or accelerated the way HTTP can. For reasons like these, Chrome 88 and Firefox 90 removed FTP support in 2021, and Debian and kernel.org retired their own FTP services in 2017. Every file we mirror is already available over HTTP and HTTPS, and most of it over rsync. More background: https://osuosl.org/blog/ftp-protocol-retirement/ If you notice any issues after the change, or have questions or concerns before then, please email support@osuosl.org.

Core IPv6 Network




Degraded Performance

Core IPv4 Network




Degraded Performance

DNS




Degraded Performance

Mail Relays




Degraded Performance

FTP Mirrors




Degraded Performance

Internal Monitoring




Operational

External Services

GitHub

Metrics

Core Network Uptime

100.00%




Core Network Response Time

116.76ms




ftp.osuosl.org Response Time

357.04ms




Core Network Uptime

100.00%




Core Network Response Time

120.50ms




ftp.osuosl.org Response Time

331.13ms




Core Network Uptime

100.00%




Core Network Response Time

97.03ms




ftp.osuosl.org Response Time

283.58ms




Locations

History (Last 7 days)

Websites slow or unavailable: shared MySQL databaseDegraded Performance

Incident Status

Degraded Performance


Components

Internal Monitoring


Locations

Salem, OR



November 8, 2026 10:59AM PST
November 8, 2026 6:59PM UTC
RESOLVED

All affected sites are back to normal. The shared MySQL database server has run normally since 10:17 AM PST (18:17 UTC), and Ogre3D's sites came back at 10:42 AM PST (18:42 UTC). The overload came from one application's database queries, which each took 16 to 47 seconds and ran many at a time until the server stopped accepting new connections. We restarted the database service, removed a server limit that let a handful of slow queries hold up every other site, and added a safeguard that stops that application's queries if they run longer than 10 seconds. If anything still isn't working, please email support@osuosl.org. We apologize for the disruption.

November 8, 2026 10:28AM PST
November 8, 2026 6:28PM UTC
MONITORING

From about 8:24 AM PST (16:24 UTC), one of our shared MySQL database servers was overloaded, so websites that use it loaded slowly or showed database errors. Other sites on our shared web hosting were also slow, because their web servers were stuck waiting on this database. Affected: - phpBB websites and services, including the forums, blog and Area51 - eLinux.org - Ogre3D's website, forum and wiki - the Amahi wiki and blog - the BusyBox bug tracker - the Mycroft Project - SeaMonkey updates - OpenID Foundation sites, the OpenPOWER Foundation's Nextcloud and our Etherpad - other websites on our shared web hosting We found the source of the load and restarted the database service. Service has been restored for most sites as of 10:17 AM PST (18:17 UTC); Ogre3D's sites are still unavailable while we work with that project. Our other database servers, mail, DNS and the FTP mirrors were not affected. We are monitoring the server and will post another update.
OpenStack Host Reboots – Thursday, Nov 5, 8:00 AM–2:00 PM PST (16:00–22:00 UTC)Planned Maintenance

Description

On Thursday we will reboot some of the servers behind our OpenStack clouds to install fixes for two Linux kernel flaws: CVE-2026-46300 ("Fragnesia") and CVE-2026-98432, the critical TCP flaw described in our separate notice on this page. Unlike Tuesday's work, the VMs on these servers can't be moved to another server while they run, so each one will be shut down and started again once, for about 20–25 minutes. Date: Thursday, November 5, 2026 Time: 8:00 AM – 2:00 PM PST (16:00 – 22:00 UTC) Affected: - Every running VM on our aarch64 (ARM64) OpenStack cloud - About 20 VMs on our x86 OpenStack cloud We are emailing the owners and members of every affected project a list of their VMs that will restart. Not affected: - All other VMs on our x86 cloud. They move live on Tuesday, November 3, with no restart (separate notice). - VMs that are shut off stay shut off. What to do: - Make sure your services start on their own when your VM boots. - These reboots update our servers, not your VM's own kernel. Please also update and reboot your VM for CVE-2026-98432 (see our separate notice on this page). If a VM doesn't come back after its restart, please email support@osuosl.org to open a ticket so we can resolve it. Questions before then can go to the same address. We apologize for the interruption.


Components

Internal Monitoring


Locations

Salem, OR


Schedule

November 5, 2026 8:00AM - November 5, 2026 2:00PM PST
November 5, 2026 4:00PM - November 5, 2026 10:00PM UTC


November 5, 2026 11:12AM PST
November 5, 2026 7:12PM UTC
UPDATE

The OpenStack host reboots are complete, and all affected VMs are running again. If a VM of yours is not back, please email support@osuosl.org.

November 5, 2026 8:14AM PST
November 5, 2026 4:14PM UTC
UPDATE

The OpenStack host reboots have started. VMs on each affected server are stopped and started again, one server at a time on each cloud.
OpenStack Live Migrations – Tuesday, Nov 3, 8:30 AM–3:00 PM PST (16:30–23:00 UTC)Planned Maintenance

Description

On Tuesday we will reboot the servers behind most of our x86 OpenStack cloud to install Linux kernel security updates. Before each server reboots, we move its running VMs to another server live, so they keep running: no restart and no reboot of your VM. Date: Tuesday, November 3, 2026 Time: 8:30 AM – 3:00 PM PST (16:30 – 23:00 UTC) Expected impact: none expected; a busy VM may pause for a second or two when it switches servers, and its network connections stay up What's happening: The updates fix CVE-2026-46300 ("Fragnesia"), an actively exploited Linux kernel flaw, and CVE-2026-98432, a critical flaw in the kernel's TCP code. Both are already mitigated on these servers, and the reboot loads the fixed kernel. We move about 260 VMs across 74 projects, one server at a time. Affected: - Running VMs on our x86 OpenStack cloud that use its shared storage, which is most x86 VMs Not affected: - VMs that are shut off stay shut off and are not moved. - VMs on our aarch64 cloud, x86 VMs on local disks, and three x86 VMs that can't be moved safely are restarted on Thursday, November 5 instead (separate notice). Their owners get a direct email listing them. You don't need to do anything. These reboots update our servers, not your VM's own kernel, so please still update and reboot your VM for CVE-2026-98432 (see our separate notice on this page). If a VM becomes unreachable during or after the window, please email support@osuosl.org to open a ticket so we can resolve it.


Components

Internal Monitoring


Locations

Salem, OR


Schedule

November 3, 2026 8:30AM - November 3, 2026 3:00PM PST
November 3, 2026 4:30PM - November 3, 2026 11:00PM UTC


November 3, 2026 1:51PM PST
November 3, 2026 9:51PM UTC
UPDATE

Today's OpenStack live migrations are complete, and all but one of the running VMs we moved did so without a restart. One VM stalled while moving and had to be restarted at 12:40 PM PST (20:40 UTC); it was back up within a minute, and we have contacted its owners directly. A few VMs lost network access for several minutes after their move, because their network port stayed attached to the old server until we moved it by hand. If a VM of yours is unreachable, please email support@osuosl.org. We apologize for the interruption.

November 3, 2026 8:02AM PST
November 3, 2026 4:02PM UTC
UPDATE

The OpenStack live migrations have started. Running VMs move between servers without a restart.