With growing demand and revenue opportunities for mobile data consumption and services, declining wireline voice revenue has become a killjoy for many operators. But at least one service provider still values wholesale residential voice over IP (VoIP) services -- as long as service quality assurance doesn't cost more than the revenue it generates. Deploying an advanced analytics engine for VoIP call detail records (CDRs) has enabled Momentum Telecom to slash its troubleshooting time, pursue new business opportunities and keep its core product lucrative.
If [VoIP quality for] a particular customer of ours has all of a sudden started degrading … we know exactly where the problem is.
Vice President of EngineeringMomentum Telecom
"You can't say you have a quality product if you can't measure it," said Anthony Orlando, vice president of engineering at Momentum, a Birmingham, Ala.-based VoIP service provider that wholesales to 230 cable operators across the country. "We had customer premise-based probes, but the analytics that came out of [them] were just not what we needed."
As a wholesale VoIP provider, Momentum's operations encompass more than its own network. When a cable operator calls with a service complaint, the underlying cause could originate in any number of sites -- Momentum's network, the cable operator's network or peering sites between providers or the public Internet.
Momentum tried homegrown and commercial VoIP analysis and monitoring tools, but the analytics were too superficial to be really useful, said Orlando, declining to name the commercial product. Network engineers often lacked the visibility to resolve issues and deliver service quality assurance, according to Alan Creighton, president and CEO of Momentum.
"[Our customers' traffic] traverses many different networks," he said. "None of that was being measured and for two years we tried to find a way to measure our service."
About a year and a half ago, Momentum received call degradation complaints from one cable operator, but the problem was unique to that customer, Creighton said. Without detailed VoIP CDR reports or sophisticated service quality assurance tools, Momentum's tier 2 and tier 3 support technicians spent two weeks trying to diagnose the issue, he said.
"Two weeks of my tier 2 and tier 3 time -- that's a lot of overhead spending on one particular customer," Creighton said. "We had audited everything on our network, everything in our [customer's], and we could not find a single thing wrong."
Momentum's engineers suspected the problem was somewhere in the cable operator's network, but couldn't prove it without the right tools and reporting engine, he said. After those two weeks, the general manager of the cable company apologized and said it had discovered the issues were related to a firmware problem with its Motorola infrastructure, Creighton said.
Reducing troubleshooting times, improving service quality assurance
For the past five years, Momentum had used Hammer XMS from Empirix, a time-division multiplexing (TDM) and VoIP monitoring and analytics tool, in its network operations center (NOC). Although it could capture and correlate signaling information to calculate mean opinion score (MOS) and R-factor reports, XMS lacked the ability to do the more sophisticated service quality assurance analysis Momentum needed, Orlando said.
After Empirix acquired VoIP monitoring vendor Mutina Technology earlier this year, it released Hammer xCentrix, which it bills as a more advanced service quality assurance analysis engine that's able to generate canned or custom reports from about two dozen key performance indicators (KPIs) from VoIP CDRs and data traffic. The bank of KPIs available for analysis is expected to grow as the product matures, according to Dan Teichman, director of product marketing at Empirix.
Momentum deployed xCentrix in its NOC in May. The analytics engine crunches data collected by XMS probes deployed in Momentum and its customers' networks.
Service providers often use CDRs -- files that contain information about the source, destination, quality and duration of calls -- for billing purposes. Empirix's software enables operators to extract and analyze VoIP CDR databases from a single dashboard without having to continually write and decipher structured query language (SQL) scripts.
Although XMS collects 1,000 KPIs from VoIP CDRs for use in 60 report templates, xCentrix offers a more powerful analytics engine for the data, Teichman said. An XMS report can tell engineers the volume of calls a probe collects over one week or the average jitter value each hour of a day, he said. The xCentrix engine can analyze multiple metrics in one report, compare them against each other and display results in a variety of graphical formats, Teichman said. Reports are also customizable, enabling engineers or executives to crunch any combination of KPIs.
"We were looking for voice quality measurements and [the ability] to slice and dice the data in multiple ways," Orlando said. "We've used this to make routing decisions … if one of our [customer's] MOS scores drop below a certain level."
Now Momentum's engineers can resolve voice quality issues typically within 15 minutes and the service quality assurance engine has "become an integral part of our day-to-day operations," Orlando said. Momentum is so confident in the VoIP CDR reports that it is opening them up to resellers and customers, Orlando said.
"If [VoIP quality for] a particular customer of ours has all of a sudden started degrading … we know exactly where the problem is," Orlando said. "Just [recently] I was on the phone with one of our carrier [customers], sent him the information and said, 'Here, you can't argue with this -- with facts. The hard cold data.'"
But service quality assurance is about more than troubleshooting, Creighton said. When cable operators can see the VoIP CDR reports from xCentrix, Creighton expects they'll entrust Momentum with wholesale enterprise VoIP services.
"For those operators [for which] we were providing [services] and had a good experience, we wanted to convince them the time was right to deploy business services products," Creighton said. "When you [operate in] a residential services market, you can have a little bit of leeway on quality. But when we introduced business products, we wanted to have the best quality … [to] gain market share in business services."
VoIP service quality assurance not for all operators
Carrier interest in VoIP monitoring and service quality assurance tools has been dwindling for years, and not just because it's been cannibalized by mobility and other services, according to telecom consultant Tom Nolle, president of CIMI Corp.
"Nobody is going to try to do assurance on individual VoIP calls because it wouldn't make any sense. It's not cost effective. The whole reason you go into VoIP is because it's dirt cheap," Nolle said. "You really can't effectively manage any kind of IP traffic efficiently and cost effectively on a per-session basis because the process is too complicated and the value of the session is too low."
As long as a network is engineered properly and infrastructure is functioning, most VoIP quality problems can be chalked up to congestion, Nolle said. Anything more complicated is unlikely to be found in VoIP CDRs, and telecom network engineers often solve issues by reconfiguring quality of service (QoS) policies, he said.
"CDRs don't carry transport detail on the individual packets or what router the call passed through," Nolle said. "There's no easy way to tell if a specific call encountered a problem…. [Operators say,] 'We're going to engineer this network for a certain probability of call problems and if the probability is below that, we're happy.'"
Voice service quality assurance products are usually unable to offer real-time analysis, he added. Most tools require technicians to attempt to duplicate the problem with a simulator, Nolle said.
Empirix's Hammer XMS product captures, stores and delivers metrics for actual sessions after they have been completed, Teichman said. Its xCentrix analysis engine is "primarily used for aggregate reporting and analysis, but does provide a capability for drill-down into Hammer XMS to see specific/individual sessions," he said.
The service quality assurance vendor pool has also been shrinking. In addition to Empirix's recent acquisition, Canadian vendor EXFO expanded its converged IP testing portfolio by acquiring Brix Networks in 2008 and its wireless testing portfolio by acquiring Nethawk earlier this year.
Meanwhile, as service providers add more capacity to their networks, "the likelihood of something as low-volume as a voice session even causing blip on the radar is zilch," Nolle added.
Let us know what you think about the story; email: Jessica Scarpati, News Writer