If you ever want to see me really swing into action, just even hint that users are having a less-than-optimal experience with their web meetings (Skype for Business, Microsoft Teams, Zoom, WebEx, Kontiki, BlueJeans...). This is the moment where I pretty much drop everything I can and log into the Zscaler admin console just as fast as I can, while simultaneously asking whoever reported it to send me the basics (client IP, approximate time, and username are generally enough to get started).
I do this because real time voice and video is really where the user experience meets the cloud. And if the users are unhappy, then senior executives at all levels get REALLY unhappy. And that is enough to prompt an immediate deep dive into the unknown.
What Goes Wrong?
Far more times than not, what I see, especially with a newer Zscaler install, are the cloud firewall configurations that aren't just quite tuned up yet. Generally this is comes in the form of the web proxy rules being in place, possibly even the network applications being defined in the firewall themselves. But then when I do the deeper dive into what traffic is being dropped for the clients, we can quickly and accurately see that other services, especially UDP, are being blocked by other rules.
A great example of this is WebEx, whereby there's a pre-defined Network Application that you can apply to a firewall rule. And it's a great start, but alone will not get you everything you need. For that we need to look at the WebEx guidance itself. On their site you will find the specific IP ranges, ports, protocols, and such to ensure that everything is added. And if you are doing regular audits of traffic that is hitting your default firewall rule, then you will likely uncover, at least on occasion, newer IP ranges as they are added by your conferencing vendors.
Anyway, before I get too far ahead, let's just recognize that what's likely going wrong is necessary traffic being needlessly dropped. And that's easy enough to fix.
Making Web Meetings Fly
Below you will see a pretty standard Zscaler firewall configuration, where I have placed WebEx right up top. I want it there because:
- I don't want someone else to come along later and convince themselves and others that we need to block any of the network services that WebEx might use (like ICMP). A well named rule like this towards the top pretty much keeps it out of harm's way.
- Web meetings are so key to business that someone may very well want to know just how much traffic goes to/from the provider. Having a single rule allows everyone to visualize that right on their dashboards, reports, log filters, Splunk, you name it.
- Honestly, I just prefer to use Network Services rather than Network Applications the defining all the traffic that needs to be allowed to known IP and FQDN groups. It makes for cleaner logs and is less reliant on deep packet inspection (DPI) signatures. So when it comes to this type of traffic, I use every little trick in the book to ensure everyone succeeds.
Beyond this, building the rule is obviously really simple. Just add the fully qualified domain names (FQDNs) and IP ranges to their own destination groups, then add those to the rule, as shown above.
TIP: To find interesting traffic for any FQDN, which can be quite helpful, just go into the Firewall Insights, Logs page and search for something like.... Client Destination Name Ends With .webex.com. That will return all the logs with anything going to something that ends with webex.com. Just export that information, run a quick pivot table on it in Excel, and in no time you will have all the FQDNs that your users are connecting to. Just do that every so often for each provider you feel you need to keep up with, and it will keep things running great. This is also one of those where no change control should even be needed, given that the application is already allowed and it is merely doing routine maintenance. In any case, let's just say that there should be no barriers to doing what needs to be done, the moment it is seen.
As noted, WebEx was just one convenient example, of which there are certainly others. Here are the top ones that I always proactively bring online and up to par with all my customers, which is to say no traffic to the providers is needlessly being dropped (click to open their tech pages):
If all the traffic is being allowed across the web proxy (URL filtering) and the cloud firewall (for all other ports and protocols), then yeah, it's time to go deeper. This means treating it as though Zscaler is the problem, up to the point that it is proven otherwise. In my experience, that is not only the best way to proceed and the way I know Zscaler themselves adheres to, but the majority of the time leads to various other root causes, such as:
- Latency / Hops: Users could be using a legacy VPN, forcing all their traffic to come back into the corporate network before going back out to the Internet.
- WAN Congestion: Be it MPLS or even local Internet breakouts, uncontrolled spikes in traffic that saturate the links can wreak havoc on voice and video traffic.
- Zscaler: Sure, it can and does happen that Zscaler will see various conditions that impact user traffic. So whenever I feel I have enough information, I first look at the Zscaler Trust site for the cloud I am on, open a case with support as fast as I can, and even look to replicate it with as many users as possible, especially those I can get to run a packet capture.
The bottom line though is that voice and video should run exceptionally well, once you have ensured that success by allowing all the traffic necessary to go out the firewall. If issues still remain, they can always be answered and remediation plans put in place, which should ultimately benefit everyone.
And if you really want your voice and video teams/owners to know you care, make sure they have your cell phone number programmed into their phone with your commitment to prioritize their needs and desires. It will pay off.