Blackboard is back up.
– Lindy Foster Enterprise Computing Oregon State University 541-737-3805
OSU suffered another border router failure between 10:36 am and 10:38 am this morning. We are in the process of shutting down the faulty device and have failed all traffic over to the standby device. This should resolve the issue until the faulty hardware has been repaired.
Good Morning,
The IS Server Support team will performing emergency maintenance on one of the Exchange servers tonight at 6pm to address remote management services issues on one of the three servers. We do not expect this maintenance to cause an outage, however users mailboxes who are associated to that server will see a two messages while their mailbox switches to the redundant server(s) and back.
Who will this affect: Mostly Transparent to customers. When will this happen: Thursday – 05-16-2013 @ 06:00PM – 6:30PM What are we doing: Rebooting the server to reset the WinRM, IIS and associated services on Why are we doing this: Troubleshooting the WinRM, and other issues related to this specific server.
If you have any questions or concerns please feel free to contact us.
Jason Appah Network Engineering/Server Support B211 Kerr Admin Bldg Oregon State University Corvallis, OR 97331 541-737-8801 Jason.appah@oregonstate.edu [Description: cid:image001.png@01CD583B.8BD4B460]
Notice of Unplanned Outage
At approx. 1:20 a.m. this morning, one of our routers rebooted. The backup router took over shortly afterward. However, there was a short disruption to internet connectivity.
We believe service is now back to normal.
If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.
Notice of Unplanned Outage
Today starting around 10:40 am lasting until approximately 11 am our authentication servers for the OSU_Secure wireless network and Enterprise VPN lost connection with its Active Directory server. This resulted in new connections to the OSU_Secure wireless network and Enterprise VPN for staff users being denied. The outage only affected users logging in with Active Directory accounts during that that timeframe. ONID authorization and the OSU_Access wireless network were unaffected.
If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.
Primary Contact: Mike Akey, mike.akey@oregonstate.edu
I will be moving some internal systems off of a network switch that needs replacement. This will result in a very brief blip in availability for the database that Maintain uses.
Cutover at: 5pm Friday 17th.
Notice of Service Interruption
We experienced a MySQL service interruption from 12:28-12:38 this afternoon.
At 12:30 MySQL failed over to the secondary server and service availability was restored. The service took approximately 8 more minutes to become fully responsive due to heavy inbound traffic.
During the failover applications connecting to the MySQL service failed resulting in a blue page (if it was a drupal site) or possibly a blank page or error page for a custom application. Once the failover was complete for the next 8 minutes applications were extremely slow in loading and customers would have noticed long page load times (5-6 seconds) on all web infrastructure that relies on the MySQL service.
This failure was caused by pushing logging configuration changes directly to the production server as opposed to the staging environment where they should have been pushed first. This is a procedural issue that will be addressed.
We apologize for the service interruption and any inconvenience it caused. If you have any questions or concerns please contact us at 7-7SIG.
Thanks, Shayne
——————————- Shayne Huddleston Shared Infrastructure Information Services 541.543.8996
Primary Contact: Josh Zojonc, Josh.Zojonc@oregonstate.edu
We will be bundling up three extremely minor well understood changes into a single maintenance action.
The first change will be to deny non-secure SSL renegotiation, to address the vulnerability described in RFC 5746. This will not have an impact on legitimate users, nor will they notice any changes.
The second change will be to replace the self-signed certificate used for logging into the management interface and changing the address (url) used for management. This change will only impact those that log into the netscaler to manage it. An email will be sent to those that will be affected.
The last change will to implement a fix for an asymmetric routing issue we’ve been dealing with. This will not have a noticeable impact on existing services or users. We have to make this change to allow further onboarding of services to the Netscaler.
In the event of any problems changes will be rolled back.
Communication will be done via http://blogs.oregonstate.edu/sig
We are currently experiencing network issues that may be related to a denial of service we experienced over the weekend. We are working on the problem now and will post more information when it becomes available.
Per our previously announced maintenance schedule, Blackboard will be unavailable on Saturday, May 18, 6:00am to 9:00am.
The complete 2013 calendar for Blackboard maintenance can be found at http://blogs.oregonstate.edu/bbsc/2013/03/22/scheduled-maintenance-2013.
– Lindy Foster Enterprise Computing Oregon State University 541-737-3805
Starting at 11:06am today Freenode received coordinated DDoS attacks causing several service outages on campus , NERO did mitigate the attack at about 12:30pm. And service now seems to be normal.
There were reported outages at: 11:06-11:18 (12 min) 11:23-11:25 (2 min) 11:31-11:37 (6 min) 11:56-11:59 (3 min) 12:01-12:04 (3 min)
-Josh
Notice of Unplanned Outage
Starting at 11:06am today OSU has been under a DDOS attack. There have been reported ongoing outages related to the attack involving Blackboard, Internet and other services. NERO is responding to the attack. We will provide further information as it is known. The service disruptions as reported at this time are:
11:06-11:18 (12 min) 11:23-11:25 (2 min) 11:31-11:37 (6 min) 11:56-11:59 (3 min) 12:01-12:04 (3 min)
At this time service seems to be normal.
If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.
The EmpCenter time and attendance system will be unavailable from 11:59 PM on Saturday May 11 through 2:00AM on Sunday May 12 due to planned firmware upgrades by the vendor.
Jill Swenson Associate Director, Enterprise Computing Services Oregon State University 541-737-0682
Primary Contact: Michael Gromek, michael.gromek@oregonstate.edu
Network Engineering will be adjusting some Internet route metrics on the active border router to ensure that maintenance while bringing the primary router back online does not cause an unintentional outage. This process is not expected to create any interruption to service, and will be performed on May 8 at 10PM. There is a small possibility that an unexpected reset could occur and cause a very short delay delay for some Internet traffic while the routing table updates. This may be noticed by monitoring tools but should not cause a loss of service.
Notice of Unplanned Outage
WebVPN and AnyConnect users were unable to establish a VPN connection after this morning\’s 6:30 A.M. reboot of the enterprise VPN. The problem was due to the AnyConnect client settings not being preserved after the reboot. The settings have been restored and WebVPN and AnyConnect users have full VPN access as of 9:30 A.M. VPN users connecting using other methods such as IPSec were not affected.
If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.
** Maintenance Announcement – No service interruption anticipated **
Services Maintained & Purpose of Maintenance: Tuesday May 7th at 6:30 am Network Services is planning to power cycle the enterprise VPN to solve a memory leak problem and prepare for an eventual code upgrade. The power cycle will take about 5 minutes and connected and newly connecting clients during that time may be disconnected or unable to login. However most users will not be affected as it will failover to the secondary VPN
Maintenance Window: Start: 05/07/2013 6:30 AM End: 05/07/2013 6:45 AM
If you have questions or concerns about this maintenance, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.
The core network suffered an hardware failure around 2 AM this morning leaving some equipment in an unstable state. This resulted in a loss of service for the residential networks from 9:43 AM to 9:55 AM this morning when the failed equipment was removed from service.
The issue has been resolved and we do not expect further disruptions due to this incident.
Tomorrow morning at approximately 0630 I will be making a configuration change to the mail relays to allow for email address mapping to be split from one file into multiples.
No outage is expected, though there is a potential for brief mail delay if I make an error.
-Mark Hammerschmith Linux Administrator 541-737-4734
An additional 2TB of disk has been allocated to CN-Share. I’ll be extending the partition during the maintenance window on Saturday may 4 after 10 PM.
Bruce Beicke Systems Administrator 541-713-3551 bruce.beicke@oregonstate.edu
A security patch has come down that I will be applying to these systems during our regular maintenance window on Saturday May 4 after 10 PM.
Bruce Beicke Systems Administrator 541-713-3551 bruce.beicke@oregonstate.edu
