Pages

Friday, December 23, 2022

Complex Systems - Part 3

As faithful blog readers will know, yours truly has been highlighting an IT problem that developed in late September and which wasn't fully resolved apparently until yesterday. We reproduce the entire sequence of emails on the subject from past blog posts to make a point. That point is that complex systems are just that - complicated. And in emergencies, such as the Big One earthquake that will someday arrive, they will fail. In such an event, your cellphone won't work. Anything related to the internet won't work. The VoIP system that is replacing the old landline phone system at UCLA won't work - regardless of the assurances you may have heard. The power will be off, as it was in the Northridge quake (which did not, however, disable the old landline phone system). 

UCLA used to have a low-powered AM radio station for emergency announcements which seems to have been abandoned. That station, so long as it had back-up power - which presumably it did, would have worked for emergency information. Note that radios - often crank operated - are commonly part of earthquake kits. Most cars also have battery-powered radios. So, why was the station abandoned?

It's all something to think about in the rubble when the Big One happens or now while you enjoy your holiday break. Here is the promised sequence:

Excerpt from first email dated September 27:

There is currently an issue with UCLA’s wireless network that is impacting the entire UCLA campus. Central Campus IT is aware of the issue and is working toward a resolution. More information will be provided as soon as it becomes available.

Excerpt from second email dated September 28:

UCLA Central Campus IT is still working with engineers to identify and resolve the intermittent connectivity issues with the campus wireless network. More information will be provided as soon as it becomes available. Thank you for your patience.

Excerpt from third email dated September 29:

Update from UCLA Central Campus IT:

eduroam IP address shortage

Members of the UCLA community reported that they were unable to connect to the eduroam network. Network Services identified that the eduroam IP address space was reaching capacity. We have expanded the IP address space to resolve this issue.

MyUCLA access on the campus residential network (i.e. The Hill)

Members of the campus residential community reported that the UCLA_WEB or UCLA_WIFI networks were experiencing difficulty when loading ucla.edu websites. A resolution has been applied and UCLA_WEB and UCLA_WIFI networks should now be able to access all ucla.edu websites.

Intermittent campus wireless network connectivity

Members of the UCLA community reported an issue on the campus wireless network where users are able to successfully connect to the wireless network and appear as connected on their device, but after a period of time websites will stop loading. The outage period has been observed as three to four minutes but can sometimes be longer. IT Services is currently testing the network to identify the root cause and resolve this issue.

Please report current campus wireless network issues (specifically date/time of outage, building/room, and the time of the class e.g. 11:00 a.m. to 11:50 a.m.) to IT Services.

Excerpt from fourth email dated September 30:

Update from UCLA Central Campus IT:

Intermittent Campus Wireless Network Connectivity

Members of the UCLA community reported an issue on the campus wireless network where users are able to successfully connect to the wireless network and appear as connected on their device, but after a period of time websites will stop loading. The outage period has been observed as three to four minutes but can sometimes be longer. Our wireless network manufacturer, Aruba, has confirmed the current issue is limited to Apple devices with the M1 processor, and there are currently no workarounds from the network side. Aruba is working with Apple toward a resolution and will continue to be involved in active troubleshooting until this issue is resolved. IT Services is also actively working to determine workarounds or mitigation measures that can be implemented.

Based on our systems logs, we can estimate that as many as 10% of campus wireless users are impacted; however, the issue remains intermittent for those users. 

Excerpt from fifth email dated October 5:

Updates from UCLA Central Campus IT:

Apple Wireless Connectivity

Following our daily call with Apple and Aruba, Apple is continuing to review logs that have been provided to them. They will also be sending an engineer to campus on Thursday to collect data to distribute to Apple engineering. Apple has informed us we should have a response from them within 24 hours. Thank you for your patience as IT Services works to resolve these issues. Please report current campus wireless network issues (specifically date/time of outage, building/room, and the time of the class e.g. 11:00 a.m. to 11:50 a.m.) to IT Services using the Wireless Network Issues Form. 

Thank you for your patience as IT Services works to resolve these issues. Please report current campus wireless network issues (specifically date/time of outage, building/room, and the time of the class e.g. 11:00 a.m. to 11:50 a.m.) to IT Services using the Wireless Network Issues Form.

Excerpt from sixth email dated October 10:

Update from UCLA Central Campus IT:

Intermittent Campus Wireless Network Connectivity

We would like to report a potential workaround to the connectivity issues. Initial testing with Bluetooth disabled seems to have relieved the wireless connectivity issue. We also noticed an increase in bandwidth with the Bluetooth radio disabled. ITS is continuing to test this but is recommending disabling Bluetooth as a workaround until a permanent solution is available.

Thank you for your patience as IT Services works to resolve these issues. Please report current campus wireless network issues (specifically date/time of outage, building/room, and the time of the class, e.g. 11:00 a.m. to 11:50 a.m.) to IT Services using the Wireless Network Issues Form.

Excerpt from seventh email dated November 17:

Thank you for your patience. Below is a summary from UCLA Central Campus IT.

Intermittent Campus Wireless Network Connectivity

Apple has confirmed that the problem is on their end.  They're working on developing a fix to push into the next code release for Ventura (13.1).

Apple has had several engineers on campus over the last few weeks testing different firmware builds / beta releases side-by-side with vanilla 13.0 as a baseline.

Apple has not shared any detailed information about the problem to date.  Engineers we've been in direct contact with have been forthcoming about what they know.  It's not clear if Apple will share root cause information or not.

Apple has pushed what they believe is a fix into one of the 13.1 beta releases (beta 2, 22C5044e) and has had positive results testing that release on campus.  Our testing shows overall performance improvements, but some testers have reported that there are still occasionally dropouts.

Until Apple has provided a permanent resolution for the issue, IT Services recommends disabling Bluetooth and set Airdrop to “Allow me to be discovered by: No One”.

Excerpt from eighth email dated December 2:

Update from UCLA Central Campus IT:

Apple Engineering has released an updated Beta Seed to address the peer-to-peer issue affecting Wi-Fi connectivity for Apple MacBook M1/M2 devices and Apple Mobiles devices.  IT Services will be assisting Apple with rigorous testing next week to provide testing results.  As we await a permanent resolution, Apple is still recommending the following workaround for Apple devices:

Disable Bluetooth

Set Airdrop to "Allow me to be discovered by: No One"

Step-by-step instructions for updating these settings can be found within our Knowledgebase.

Excerpt from ninth email dated December 12

Happy holidays and I hope you have a wonderful break.

Below is an update from UCLA Central Campus IT:

Apple has promoted the latest macOS Beta to a Release Candidate, addressing the peer-to-peer issue affecting Wi-Fi connectivity for Apple MacBook M1/M2 devices and Apple Mobiles devices.  All testing with both the Beta and Release Candidate has been positive and IT Services, Apple, and other campus entities have not been able to reproduce the issue.  We will provide another update once we are notified of an official release of macOS 13.1.

Please note, we have been made aware that the official resolution will only be addressed in macOS Ventura.

And finally, excerpt from email dated December 22:

Dear Bruin Community:

Thank you for your patience as we’ve worked through issues affecting campus Wi-Fi connectivity on some Apple devices.

IT Services is pleased to share that Apple has confirmed the release of an update to address the peer-to-peer issue affecting Wi-Fi connectivity for Apple MacBook M1/M2 devices and Apple Mobile devices. We apologize that it has taken so long to provide resolution on this issue with Apple. 

To resolve the issue, you will need to update your device as follows:

• Update your Mac to macOS Ventura 13.1.

• Update your iPhone or iPad to iOS/iPadOS 16.2.

If you are unable to update to macOS Ventura 13.1 or iOS/iPadOS 16.2 due to software or hardware incompatibility, you can install Security Updates for your current OS to address the Wi-Fi connectivity issue. 

For more details regarding Operating System versions that address the Wi-Fi connectivity issue or assistance with updating your device, please contact the IT Support Center.

===
To hear the text above in a computer generated voice that will struggle with some of the jargon, click on the link below:

https://ia601402.us.archive.org/25/items/big-ten/final%20resolution.mp3

No comments: