Overview
The Sophos UTM v9.703 release was temporarily pulled and re-released on 23 April 2020.
Sophos received reports from a subset of Sophos UTM v9.703 systems, where the update has caused issues with GUI access and traffic passing through the firewall.
Applies to the following Sophos product(s) and version(s)
Sophos UTM v9.703
Current Status
UTM v9.7 MR3 was re-released on, 23 April 2020.
More info available in this Community post
Timeline
On Apr 8 2020, SG UTM v9.703, which is a regular maintenance release, was made available as a "soft-release" to customers (this means the update was available for download manually, but not pushed as an update to UTMs). This soft-release period lasted for a week, with ~5600 UTMs installing v9.703 during this time. Even though a few users reported an issue with this update on the UTM community forum on Apr 11, v9.703 was released generally and pushed as an update to all UTMs on Apr 15 2020. On Apr 16 2020, the 9.703 update was removed from the UTM update channel.
Issue Detail
One of the bug fixes included in v9.703 was to fix an issue where site-to-site IPSec tunnels may not be established properly when DHCP is configured on the WAN interface. The issue happens when the UTM tries to establish the IPSec tunnel before the interface receives its IP address & gateway through DHCP. A fix was implemented to have the UTM check for DHCP address, gateway & required routes before trying to establish the IPSec tunnel.
Unfortunately the fix did not take into account Remote Access IPSec. Remote Access IPSec objects in the UTM do not contain a gateway, and the fix does not handle this properly. This means if Remote Access IPSec is configured/enabled on the UTM, after the UTM upgrades to v9.703, middleware goes into a failed state and interrupts traffic flowing through the system.
Issue Impact
Out of ~5600 UTMs which upgraded to v9.703, 4 customers reached out to Sophos Support to report they were affected by this issue.
Root cause
Inadequate testing - Due to a communication gap within the Engineering group, the fix for the DHCP/site-to-site IPSec issue did not get communicated to the team responsible for Remote Access. This resulted in the Remote Access tests not being executed on v9.703 as they should have, missing this regression and allowing the problem to escape into the field. We have identified & addressed this communication gap, and will also be reviewing/adjusting the release tests which get run for every release to ensure the most common deployment scenarios are covered.
Responsiveness - Due to a miscommunication, even though some users reported this issue on the UTM community forum on Apr 11, Global Escalation Support & Engineering did not get notified of the issue until Apr 16. This resulted in v9.703 being made available generally, exposing more UTMs than necessary to this issue. The escalation process between the forum moderators, Support & Engineering is currently being reviewed.