Central Web Services

Notice of Unplanned Outage

While doing planned maintenance on the b210 nexus gear there was a 1 minute unplanned outage while vpc re-convergence took place.

The nexus gear in a016 had some packet loss during maintenance. It could have been enough to be spotted by monitoring systems, but should not have caused any noticeable outage by users.

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Beginning Tuesday, Feb. 14 Central Web Services will begin upgrading hosted sites to the latest release of OSU Drupal. This release includes a some important security updates, as well as a new “feature” for easily displaying videos on your site. Please read the announcement for full details of what is included in this release.

The schedule for updating your sites is as follows:

Tuesday Feb 14th 8:30 AM to 5:00 PM – Update development (drupaldev) sites Wednesday Feb 15th 8:30 AM to 12:00 PM – Update group and organization (VSO and SSO) development sites.

Tuesday Feb 21th 8:30 AM to 5:00 PM – Update production sites. Wednesday Feb 22st 8:30 AM to 12:00 PM – Update group and organization (VSO and SSO) production sites.

This is a rolling update and we cannot say specifically when each site will be updated. The update takes about 5 minutes for each site during which a visitor to the site may see the maintenance screen displaying your site maintenance message. You can view or change your site maintenance message by navigating to Site Configuration -> Site Maintenance when logged in as a site administrator.

Please contact us if you have any questions or concerns.

— Paul Lieberman Lead Web Developer Central Web Services

Service(s) affected: Wireless network access, Withycombe 1st floor

Configuration change to provide enhanced access to a single access point on the first floor of the Withycombe building.

Maintenance Window: Start: 02/10/2012 7:00 AM End: 02/10/2012 8:00 AM

Reason for maintenance: Customer request to add connectivity for new devices.

Impact:

Locations affected: Withycombe 1st floor

Applications affected: None

If you have questions or concerns about this maintenance, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Service(s) affected: We do not anticipate any outage. But Data center connectivity for servers in Milne, A016, and B210 could potentially be affected.

Each nexus device will be rebooted one at a time.

Maintenance Window: Start: 02/11/2012 10:00 PM End: 02/12/2012 6:00 AM

Reason for maintenance: After the change there will be a predetermined primary device.

Impact:

Locations affected: data centers Milne, Kerr B210 and A016.

If you have questions or concerns about this maintenance, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

All,

Tonight I will be installing the php5 memcache and apc modules on our production webheads. This should give us a significant improvment in capacity.

This should now cause any outages and should be transparent too users.

Please let me know if you have any questions or concerns.

Josh Zojonc Central Web Services

Service(s) affected: Mysql and MongoDB

Need to take the time to upgrade the servers to debian squeeze as well as fix MySQL replication

Outage Window: Start: 02/11/2012 9:00 PM End: 02/11/2012 11:00 PM

Reason for outage: Upgrading OS and fixing MySQL replication

Impact:

Applications affected: Maintain, BanHammer, Greylisting

Network Outage Policy http://oregonstate.edu/net/info/policy/outagepolicy.php

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Blackboard experienced database connection issues which resulted in an interruption of service from approximately 6:30pm to 7:50pm.

Service was restored at 7:50pm and we are investigating the cause of the database issue. –

Stacy Brock Enterprise Computing Oregon State University

Blackboard is currently experiencing db connection issues that are affecting all our app servers.

We are aware of the issues and are working to restore service as soon as possible. –

Stacy Brock Enterprise Computing Oregon State University

Notice of Unplanned Outage

During the planned maintenance window that was not supposed to cause an outage did indeed cause an outage that lasted 5 minutes. The fail-over did not occur.

I\’ll follow up with what went wrong. If there are any questions or concerns please contact us.

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Service(s) affected: Servers in data centers Milne, A016, and B210

Reboot of redundant switches.

Maintenance Window: Start: 02/02/2012 5:45 PM End: 02/02/2012 7:45 PM

Reason for maintenance: Currently the primary switch is determined by lowest MAC address. After the change the primary switch will be chosen via a setting called VPC system-priority. Basically, I’ll choose the switch for primary.

Impact:

Departments affected: None

Locations affected: None

Applications affected: None

If you have questions or concerns about this maintenance, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

All,

Our BL490s have an hotfix for the XenServer network driver. There should be no outages as a result. VMs will be moved around as this hotfix is applied to the hypervisors themselves.

This maintenance will take place 2/4/2012 at 2200 (10pm) and will take 3 hours to complete.

If you have any questions feel free to contact me.

-Joseph

_________________________________________________________________________ Info: Contact Information: Joseph T. Duncan email: duncanjo@ucs.orst.edu Student Computing Facilities phone: 541-737-2556

On Thursday Feb 2nd at 7am I will be making some settings changes to the Cisco Clean Access Manager. This is the device that helps control access to the wireless network.

In making these changed there is the possibility that wireless access will be affected. I do not anticipate an outage of more than a minute or so, if any outage does occur. In the case of a prolonged outage we have a plan in place to bypass the management device to allow wireless clients to access the network.

If you have any questions please contact Alec Dhuse (alec.dhuse@oregonstate.edu)

Notice of Unplanned Outage

There appeared to be a DHCP problem on the subnet that was added to the OSU_Secure wireless network (yesterday). We believe that the cause of the problem has been identified and remedied.

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Service(s) affected: All servers connected to b210nxs2

Eth100/1/1 BS-Avanti #D.001 D connected 483 full 1000 — Eth100/1/2 #das1.tss.oregonst connected 483 full 1000 — Eth100/1/3 bs-web #Dell36-008 connected 483 full 100 — Eth100/1/4 #oobwin2; dell37-0 connected 20 full 1000 — Eth100/1/5 #ir-sql.tss.oregon connected 483 full 1000 — Eth100/1/6 #ous-teammate.tss. connected 483 full 1000 — Eth100/1/7 #ext-pachyderm.tss connected 483 full 1000 — Eth100/1/8 #tss1.tss.oregonst connected 732 full 1000 — Eth100/1/9 #cnlite2.tss.orego connected 272 full 1000 — Eth100/1/13 #xen2.cws.oregonst connected 479 full 1000 — Eth100/1/14 #xen2.cws.oregonst connected trunk full 1000 — Eth100/1/15 #xen3.cws.oregonst connected 479 full 1000 — Eth100/1/16 #xen3.cws.oregonst connected trunk full 1000 — Eth100/1/34 #droopy.nws.oregon connected 498 full 1000 — Eth100/1/35 #droopy-private; D connected 3820 full 1000 — Eth100/1/36 #cerberus-private; connected 3820 full 1000 — Eth100/1/37 #cerberus; dell56- connected 498 full 1000 — Eth100/1/38 #milo; dell56-03 connected 6 full 1000 — Eth100/1/39 astro #dell56-04 connected 498 full 1000 — Eth100/1/40 astro-NET_Logging- connected trunk full 1000 — Eth100/1/42 #oob.cws dell33-48 connected 12 full 1000 — Eth100/1/47 #bs-vm1a dell36-11 connected 483 full 1000 — Eth100/1/48 #bs-vm dell36-09 connected 465 full 1000 — Eth110/1/1 # ldap4.onid.orego connected 20 full 1000 — Eth110/1/2 # ex2.oregonstate. connected 514 full 1000 — Eth110/1/25 #airwave-new.nws.o connected 498 full 1000 — Eth110/1/26 # ldap5.onid.orego connected 20

This device will be reloaded.

Maintenance Window: Start: 02/01/2012 6:30 AM End: 02/01/2012 8:00 AM

Reason for maintenance: b210nxs2 is the primary VPC, which means it’s the active switch. I need to repatch some fiber. So I want it to be the secondary VPC.

Impact:

Departments affected: This should affect no affect.

Locations affected: No locations should be affected.

Applications affected: No applications should be affected.

If you have questions or concerns about this maintenance, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

Network Services needs to move the Modular Messaging Servers to a new rack location to accommodate the planned installation of new core network equipment. We will perform the move in the morning of Wednesday, February 1st. The move window is from 5AM to 7AM. During this time the voice mail service will not be available to receive calls. All service is expected to be restored by 7AM. This outage will affect Corvallis, Cascades and HMSC.

_______________________________________________________________________________ [cid:image001.png@01CCDC43.EF280A40]

NERO, our upstream Internet provider, has provided the following information about the outage we just experienced:

“The problems occurring on the Pittock Core router this morning began affecting eugn-core1-gw this afternoon beginning around 1519. We have reloaded the linecards on that router and traffic has returned to normal as of 1555.

NERO will add eugn-core1-gw to the list of routers needing maintenance during tomorrow morning’s emergency window.”

If you have questions or concerns about this outage, please email net@oregonstate.edu.

-Kirsten Petersen Network Services Oregon State University http://oregonstate.edu/net net@oregonstate.edu

We are seeing a repeat of the network connectivity issues we had this morning. We will send more information as it becomes available.

-Kirsten Petersen Network Services Oregon State University http://oregonstate.edu/net net@oregonstate.edu

The Blackboard system will be unavailable from 4-7am on Saturday January 28th for maintenance.

Thank you

– Frank Kessel Enterprise Computing support@my.oregonstate.edu

Notice of Unplanned Outage

There have been reports of intermittent Internet connectivity issues. We are investigating the cause.

If you have questions or concerns about this outage, please contact Network Engineering at net (at) oregonstate.edu or call 737-HELP.

All,

Tonight at 10pm I will be moving the VIP (virtual ip) of our mysql5 development cluster (mysql5-dev.cws.o.e) onto a new machine. This move is being made to increase the disk space of the cluster.

There will be a short interruption of service while the ip is brought down on the current machine and brought up on the new. It should be less than a minute. It will not affect any production applications or services.

Let me know if you have any questions or concerns.

-Josh Zojonc Central Web Services