Veteran Microsoft engineer shares some enterprise support tips

Microsoft veteran Raymond Chen revealed another product support trick from within the corridors of Microsoft. This time, it's not about blowing on connectors but about avoiding casting some embarrassing shade on a customer's purchasing decisions.

When something goes wrong in the Enterprise, it isn't always DNS. It might have something to do with something a customer has installed - perhaps a driver, or maybe that anti-malware software that has latched on to something deep in the internals of Windows and is now wreaking havoc.

Chen explained, "It can happen that investigating a problem reveals that a problem occurred when calling a function that has been patched or hooked."

He added, "In the case of enterprise customers, the offender is typically some 'advanced anti-malware software' that they paid a lot of money for."

"The code running in the hook ends up does something sketchy, the most common example of which is hooking a low-level function and then having the hook call a higher-level function, resulting in a deadlock."

Chen used the example of hooking HeapAlloc, a low-level function, and calling MessageBox, a high-level user interface function. He called his example "ridiculous," although this writer would counter with "let those who have not done something similarly naughty in the name of debugging cast the first stone."

The problem is working out a way to let the customer know that it is the vendor's software that has caused the problem. "After all, they paid a lot of money for that anti-malware software, and a recommendation of the form 'throw away a lot of money you already spent' is not going to land well," said Chen.

Instead, the trick is to persuade the customer to turn off the offending anti-malware software to… er… better debug the issue. In other words, you tell the user, "We're not smart enough to debug the problem. Can you help us?"

"I'm told that what usually happens is that the customer, for some mysterious reason, is unable to get the problem to occur when the anti-malware software is disabled. 'Wow, that's weird'."

• PEBCAK problem transformed young techie into grizzled cynical sysadmin
• VirusTotal: We're sorry someone fat-fingered and exposed 5,600 users
• Errors logged as 'nut loose on the keyboard' were – ahem – not a hardware problem
• That hardware will be more reliable if you stop stabbing it all day

Chen said, "Sometimes the customer gets the hint and opens a support ticket with the anti-malware vendor.

"Sometimes we have to suggest to them, 'Why don't you check if there’s an update available for your anti-malware software?'"

And the blowing on the connector trick? Twenty years ago, Chen explained that enterprise support engineers would ask customers to blow dust out of connectors rather than suggesting that perhaps, just perhaps, they'd forgotten to plug something in or plugged it into the wrong port.

Now, just like then, the move allowed the customer to save face while also fixing the problem, which some might say is the true craft of a quick-witted support engineer. ®