Central Web Services

This morning at 7:35 we had another unexpected reboot of an Exchange server and the databases successfully failed over to the remaining servers. Our core dump analysis indicates that the failure was in the TCP/IP Driver. We have reviewed all the updates that were applied on 3/31/2013 and only one update (KB2790655) modified this driver. We have uninstalled the updated driver from all Exchange servers and they are pending reboot. We plan to reboot the Exchange systems tonight starting at 9:30 unless they expectedly reboot prior to this time. The primary impact of these reboots will be the disconnection of OWA sessions.

Cary Shufelt

We had another unexpected reboot an Exchange server at 7:30 this evening. As an emergency measure, I am uninstalling Exchange update that was applied to the Exchange servers yesterday. This will result in unexpected disconnections for users using OWA.

Cary

(My working hours are 7am – 5:30 pm, Tuesday – Friday) ———————————————————- Cary Shufelt, MCSA, MCSE Windows Infrastructure Architect Network Engineering B211 Kerr Admin Bldg Oregon State University Corvallis, OR 97331 541-737-9420

One of the redundant exchange servers unexpectedly rebooted at 2:00pm today. This appears to be related to a potential denial of service, denoted by a spike in traffic to the server. no conclusions have been made yet We believe the only user impact would be that users connected via OWA to ex3 may have been disconnected. All services are up on the redundant servers and we will redistribute after close of business today.

Jason Appah Network Engineering/Server Support B211 Kerr Admin Bldg Oregon State University Corvallis, OR 97331 541-737-8801 Jason.appah@oregonstate.edu [Description: cid:image001.png@01CD583B.8BD4B460]

We experienced a service interruption to Exchange this morning during our routine maintenance. Between 10:36 and 10:47AM users could not establish new connections to Exchange OWA and the POP3 service was down from 10:34 to 11:15AM. It appears that both of these outages were the result of configuration changes on the NetScaler load balancer.

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Cary Shufelt

Notice of Unplanned Outage

At approximately 6:58 am the connection to the b210 data center in Kerr Administration building went down for about 30 seconds due to an adverse affect of routine maintenance. We are investigating the cause and will evaluate if the maintenance performed should still be considered routine.

Joel Burks

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

The maintenance on the network device has been successfully completed and all building links are back online.

Sent: Wednesday, March 20, 2013 2:44 PM Cc: Outages and Announcements

Network Services will be performing maintenance on the ResNet networks (both wired and wireless service) on the morning of Thursday, March 28th between 5:30 and 6:30 am. The network in residence halls & cooperative houses will be down for most of the hour while we upgrade the router’s chassis and make a configuration change to resolve a performance issue they’re currently encountering.

Tony Brock Network Services, Engineering Team

One of OSU’s Internet providers, NERO (Network for Education and Research in Oregon), will be performing an upgrade to their Corvallis hub between 4:00 am and 6:00 am on Thursday, March 28. Although OSU is connected to the Internet via several providers, many external customers will experience up to two service interruptions with OSU-hosted websites and applications, with interruptions lasting between 3 and 10 minutes.

Likewise, Internet connections from the Corvallis campus may also be interrupted. However, NERO’s maintenance will not affect internal OSU services. This means that VoIP, building controls, and other on-campus systems will continue to function normally unless they depend on cloud-hosted services.

Primary Contact: Josh Zojonc, josh.zojonc@oregonstate.edu

On Saturday the 30th @10pm we will be upgrading our Netscaler load balancers from version 9.3 to version 10.0. We will be following the Citrix documentation on how to do this. The appliances will be upgraded one at a time and no downtime or outages are expected. This change will be invisible to users. In the event of any problems we will roll the changes back following Citrix documentation.

General Communication is via: http://blogs.oregonstate.edu/sig/2013/03/26/netscaler-upgrade-to-version-10/ I will be communicating with those that are able to log into the Netscaler on an individual basis.

We will be making two wireless network changes at around 6:30 a.m. on Tuesday 26th March 2013. Both have been tested under realistic load, in the Memorial Union. Neither seemed to disrupt service.

The OSU_Secure wireless network change will be the more noticeable of the two changes. It will result in OSU_Secure users receiving new I.P. addresses (see below for the details).

On campus view:

Off campus view:-

128.193.152.160 through 128.193.152.174

If you maintain an on-campus firewall (or access control list), that allows users of the OSU_Secure wireless network to get inside, you will have to update it with the new (on-campus) address ranges. If you collaborate with an outside organization, and they have allow users of the OSU_Secure wireless network to connect through their firewall, they will have to update their firewall with the new (off-campus) address ranges. If you experience a problem connecting to a resource, after we make the change, please check your firewall/access-list first.

Ian R. Downie

MyDegrees has been returned to service.

Catherine

Catherine Williams Associate Director, Enterprise Computing Services Oregon State University 224B Milne Computer Center Corvallis OR 97331 541-737-2718

MyDegrees is currently unavailable. OSU staff members and the vendor are working on a solution.

Network Services will be performing maintenance on the ResNet networks (both wired and wireless service) on the morning of Thursday, March 28th between 5:30 and 6:30 am. The network in residence halls & cooperative houses will be down for most of the hour while we upgrade the router’s chassis and make a configuration change to resolve a performance issue they’re currently encountering.

Tony Brock
Network Services, Engineering Team

Correction – Date was a month off. Sorry.

On Thursday morning 3/21 at 8 AM CWS will be updating CWS Hosted Drupal 7 sites to the latest version of Drupal. This will only affect Drupal 7 production sites. Although this will incur a 2 to 3 minute interruption in service to the affected websites, most site visitors will not notice the interruption, as most pages are cached. This will be a rolling upgrade and we cannot say exactly when or on which day a specific site will be affected.

If you have any questions or concerns please contact us through our form at http://oregonstate.edu/is/services/cws/contact.

Thank you,

Paul Lieberman Lead Web Developer Central Web Services

On Thursday morning 2/21 at 8 AM CWS will be updating CWS Hosted Drupal 7 sites to the latest version of Drupal. This will only affect Drupal 7 production sites. Although this will incur a 2 to 3 minute interruption in service to the affected websites, most site visitors will not notice the interruption, as most pages are cached. This will be a rolling upgrade and we cannot say exactly when or on which day a specific site will be affected.

If you have any questions or concerns please contact us through our form at http://oregonstate.edu/is/services/cws/contact.

Thank you,

Paul Lieberman Lead Web Developer Central Web Services

I’ll be patching CN-Home, CN-Share, TSS-Data, TSS1, and TSS4 on Saturday, March 16 during are maintenance window starting at 10 PM. Barring any unforeseen problems, patching should be complete within one hour.

Bruce Beicke Systems Administrator 541-713-3551 bruce.beicke@oregonstate.edu

Banner will be unavailable at midnight this Friday, March 9th, for our usual backups. Banner generally returns to service around 5:00am on Saturday morning. This Saturday, March 16th, Banner will remain unavailable for four more hours, and return to service at 9:00am.

Jill

Jill Swenson Associate Director, Enterprise Computing Services Oregon State University

Banner will be unavailable at midnight this Friday, March 9th, for our usual backups. Banner generally returns to service around 5:00am on Saturday morning. This Saturday, March 9th, Banner will remain unavailable for three more hours, and return to service at 8:00am.

Jill

Jill Swenson Associate Director, Enterprise Computing Services Oregon State University

Primary Contact: Jill Swenson, jill.swenson@oregonstate.edu

We will extend the Banner backup period on Saturday March 9 and Saturday March 16 to 8:00AM to load tables into the Operational Data Store. This is a well tested known activity that needs to be performed when the database is relatively quiet since it involves a number of table locks and creates moderate system load.

Email has been sent to the Banner technical teams, and business calendars have been checked to confirm that this will not disrupt planned business activities.

Starting at 03/05/2013 – 3:15am – 3:27am, an incident occurred with Core Network. This incident is currently resolved.

At approximately 3:15am today (3/5) many OSU network services became unavailable. The issue resolved itself by 3:27am. We are still investigating the cause of this outage.

Customers using mail.oregonstate.edu to send mail from off-campus computers may have had problems sending between 1:36PM and 3:32PM today. Some customers reported seeing an error from their email client about TLS not being supported by the server.

This issue was due to an incorrect configuration change that was made to try to resolve today’s issue with mail sent from OSU web forms.

We apologize for this disruption in service and will review our processes for incident management to prevent future such issues.