8.1.20 Release Notes

Fixed an issue where the traffic count displayed in the Controller interface could be inaccurate when using Distributed Cloud Firewall (DCF) with external groups that include multiple IP ranges.

The Controller now reports traffic statistics correctly for DCF rules involving external groups, providing accurate visibility for monitoring, analysis, and validation of firewall policy behavior.

Resolved an issue where Edge gateway upgrades from version 8.1 to 8.1.10 could cause temporary traffic disruption due to service restarts during the upgrade process.

The upgrade workflow now handles service restarts more effectively, reducing traffic impact during Edge gateway upgrades, including in large-scale deployments.

Resolved an issue where the ESTABLISHED rule disappeared after a Public Subnet Filtering (PSF) gateway image upgrade.

This issue affected PSF gateways using the legacy stateful firewall on Controller versions 7.1 and later, and could result in traffic disruption after the upgrade. The rule is now preserved during PSF gateway image upgrades.

Fixed an issue with database schema type definitions that could trigger migration errors during the Controller upgrade process.

The schema now uses the correct database type definition, ensuring compatibility with migration logic and preventing upgrade failures.

Fixed an issue where FireNet deployments with bootstrap enabled could fail in Google Cloud due to changes in how GCP credentials were handled during the bootstrap process.

The bootstrap workflow has been updated to correctly retrieve and use GCP credentials, ensuring FireNet deployments with bootstrap complete successfully in Google Cloud environments.

Fixed an issue where the default access control rules did not properly allow ICMP traffic used for debugging. The updated rules ensure ICMP-based troubleshooting continues to work after upgrades.

Resolved an issue where upgrading to Controller 8.1 failed during database migration if VPC tunnel records contained non-numeric values in the timestamp field.

The migration logic now correctly handles timestamp values, preventing conversion errors and allowing the upgrade to complete successfully.

Resolved an issue where eBPF packet marking could fail on transit gateways with Network Segmentation enabled, causing traffic to be associated with incorrect network domains.

The packet marking logic has been corrected to ensure Network Segmentation policies are enforced consistently without requiring service restarts.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround:

None

Recommendations:

In Controller release 8.0, gateway software upgrades take longer to complete compared to earlier versions. On average, the upgrade rate drops from approximately 14 gateways per minute in version 7.2 to approximately 11 gateways per minute in 8.0, which is an increase of about 20% in execution time.

Affected Scenarios:

Impact:

Only the upgrade duration is affected. Gateway functionality remains unaffected after a successful upgrade.

Recommendations:

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Workaround:

In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes.

Impact:

Workaround:

Contact Aviatrix Support for manual correction.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Impact:

Workaround:

When DCF processes high volumes of logging messages, rsyslogd rate-limiting may cause message loss. The system drops messages exceeding 500 per 5-second interval, with rsyslogd logging "messages lost due to rate-limiting" notifications.

Affected Scenario: High-traffic environments generating intensive logging activity

Impact:

Workaround: Monitor rsyslogd logs for rate-limiting messages and consider implementing log aggregation strategies to distribute message processing load across multiple collection points.

In Oracle Cloud Infrastructure (OCI) environments, OpenVPN clients cannot connect to VPN gateways configured with DUO multi-factor authentication (MFA). Connection attempts fail with ECONNREFUSED errors during tunnel establishment, preventing authentication from completing.

Impact:

Workaround:

No current workaround. Users may temporarily switch to OKTA or LDAP authentication if feasible.

When upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading "Service temporarily unavailable" error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes.

Impact:

Workaround:

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact:

Workaround:

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

In some cases, users later reappear as active but still show no profile association in the UI. This results in a display inconsistency between the UI and the backend state.

Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment.

Workaround: None.

When a Controller experiences out-of-memory conditions followed by upsizing and restart, duplicate resource ID entries may be created in the database. This prevents the Controller from starting properly and blocks access to the web UI.

Impact:

Affected Scenario: Controllers that have experienced memory issues, been upsized, and restarted may encounter this database corruption.

Workaround: Connect directly to the Controller database and manually remove the duplicate resource ID entries to restore normal operation.

When DPI/IDS or Layer7 policies are configured with "Destination: Anywhere" on HA-enabled spoke gateways where the destination smart group contains private CIDRs, the policies become invalid and cause traffic drops.

Affected Scenario: Spoke gateways with HA enabled using DPI/IDS or Layer7 policies that have destination smart groups containing private CIDR ranges and "Destination: Anywhere" configuration.

Impact:

Workaround: Modify the policy destination from "Anywhere" to specific target destinations that exclude conflicting private CIDR ranges, or disable HA on the affected spoke gateway if operationally acceptable.

When clients use HTTP/2 protocol, Trafficserver incorrectly reuses origin connections, which can cause SSL/TLS verification issues and potential security concerns with SNI (Server Name Indication) handling.

Affected Scenario: HTTP/2 client connections through Trafficserver proxy

Impact:

Workaround: Configure records.yaml to match on both IP address and SNI to ensure proper connection handling.

When a gateway is downsized to a smaller instance type, L7 traffic inspection policies cannot be enforced due to insufficient resources, causing HTTP/HTTPS traffic to be blocked instead of allowed through. The traffic-server service fails to run on the downsized gateway, triggering a DCF (Data Control Framework) error notification indicating the gateway cannot handle the configured policies.

Impact:

Affected Scenario: Gateways with L7 inspection enabled that are downsized to instance types with insufficient memory or CPU resources.

Workaround: Resize the gateway back to an instance type that meets the minimum resource requirements for L7 inspection, or temporarily disable L7 policies if downsizing is required.

In some environments, after the Identity Provider (IdP) rotates its SAML signing certificate, the Aviatrix Controller may fail to fetch and update the new certificate from the configured metadata URL.

As a result, the Controller continues to use a stale certificate, which causes signature verification errors during SAML authentication.

Impact: SAML single sign-on (SSO) authentication fails. Users may experience repeated login failures or timeouts and are unable to access the Controller dashboard using SAML.

Workaround: Contact Aviatrix Support to manually update the SAML certificate on the Controller.

When upgrading gateway software from version 7.2 to 8.0.30, the VRRP state file becomes empty on AEP edge gateways configured in active-active HA pairs. The keepalived service continues running and the keepalived.conf file retains proper configuration, but /etc/localgateway/vrrp_state.json loses its primary/backup state information.

Impact:

Affected Scenario: AEP edge gateways with VRRP configuration during software upgrade from 7.2.x to 8.0.30

Workaround: Reconfigure VRRP settings on affected gateways after the upgrade to repopulate the state file.

When the Controller processes inventory data across multiple accounts and inventory types, the public.inventory table in the database continuously grows by inserting new entries for each inventory operation instead of updating existing records. This results in unnecessary database bloat with potentially millions of redundant rows.

Impact:

Workaround: Monitor database size regularly and consider periodic cleanup of old inventory entries through database maintenance windows until the fix is implemented.

When CoPilot Asset Inventory (CAI) performs queries on the inventory table, the existing database indexes are not utilized effectively, causing performance degradation during inventory operations.

Affected Scenario: CAI inventory queries for retrieving resource counts and metadata across cloud service providers experience slower response times.

Impact:

Workaround: None.

When upgrading the Controller to version 8.1, the database migration may fail if the tunnel rtt_avg field contains None values. The migration logic expects either a numeric value or the string "N/A", and encountering a None value causes the upgrade to stop.

Impact:

Workaround: Contact Aviatrix Support for assistance in correcting the database values before retrying the upgrade.

Azure controllers using default P6 disk tier (240 IOPS) may experience performance issues, particularly with 8.x containerized controllers. This limitation can cause system instability and processing delays during high I/O operations.

Affected Scenario: Controllers launched from Azure marketplace AMI with default disk configuration.

Impact:

Workaround: Upgrade the Azure controller disk tier from default P6 (240 IOPS) to minimum P10 (500 IOPS) through Azure portal disk configuration settings.

When ICMP traffic passes through Suricata inspection on gateways, alert rules trigger only once until the Suricata process restarts. This limitation affects ICMP and potentially other non-TCP traffic using the eBPF → proxyPcap → Suricata inspection path.

Impact:

Affected Scenario: Gateways with Suricata-based threat detection enabled processing ICMP or UDP traffic.

Workaround: Contact Aviatrix Support to restore the alert functionality for ICMP traffic.

When processing decrypted POST traffic through the ATS tee plugin, PSF gateways may experience crashes during request body processing. This occurs specifically with decrypted traffic that contains POST requests being processed by the tee plugin’s request body handling path.

Impact:

Affected Scenario: PSF gateways processing decrypted POST traffic through ATS tee plugin

Workaround: Avoid routing decrypted POST traffic through affected PSF gateways until the fix is implemented. Consider using alternative routing paths or temporarily disabling tee plugin functionality for POST request processing if operationally feasible.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Affected Scenario: Load balancer–enabled VPN gateway deployments on Controller versions 8.0, 8.1, and 8.2.

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

In Aviatrix software versions 8.1.x and 8.2.0, the VRRP state file /etc/localgateway/vrrp_state.json, may be empty on AEP and self-managed Edge-as-Spoke gateways configured in active-active HA pairs. This prevents VRRP state updates from being sent from the edge gateways to the Aviatrix Controller, and Aviatrix CoPilot will not display the updated VRRP states. This is a cosmetic issue and there will be no disruption to traffic.

Impact:

Affected Scenario: - AEP and self-managed Edge-as-Spoke gateways in active-active HA deployments with VRRP enabled upgrade from 8.0 to 8.1 or created in 8.1 and 8.2

Workaround: Please contact Aviatrix Support for assistance to help you fix the incorrect display of VRRP states in Aviatrix CoPilot.