Exceeded allowed rate for Reatime information
Description
Imagicle UCX Suite syncs it's data about phones and, if configured, users with the CUCM cluster using a mechanism known as AXL which is an XML/SOAP API endpoint provided by the CCM platform.
The API endpoint is limited, by default, to 15 queries per minute and have a maximum configurable value of 18 under the CUCM Enterprise Parameters for Rate Control → Allowed Device Queries Per Minute
The AXL client used in the IAS can exceed this rate and will log under %IAS_ROOT%\Var|Log\SvSasAXL.log_<date>.txt an error such as (assuming the debug flag under Support -> Logs is set):
[DEBUG 1] <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode xmlns:ns1="http://xml.apache.org/axis/">ns1:Server.RateControl</faultcode><faultstring>AxisFault: Exceeded allowed rate for Reatime information. Current allowed rate for realtime information is 15 requests per minute.http://schemas.cisco.com/ast/soap/action/#RisPort#SelectCmDevice</faultstring><detail><ns2:hostname xmlns:ns2="http://xml.apache.org/axis/">DCEN-0VAPL00IP</ns2:hostname></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>
00:01:47 - 00.0000 [DEBUG 1] This is a SOAP response
00:01:47 - SOAP RESPONSE contains ERROR: AxisFault: Exceeded allowed rate for Reatime information. Current allowed rate for realtime information is 15 requests per minute.http://schemas.cisco.com/ast/soap/action/#RisPort#SelectCmDevice
00:01:47 - GetAXLData
00:01:47 - 00.0030 [INFO 2] [2022-01-31 00:01:47] AxisFault: Exceeded allowed rate for Reatime information. Current allowed rate for realtime information is 15 requests per minute.http://schemas.cisco.com/ast/soap/action/#RisPort#SelectCmDevice
00:01:47 - 00.0020 [INFO 2] [2022-01-31 00:01:47] AXL Query got SOAP error response
00:01:47 - *Detected a persistent soap error in phase [Device Details] while importing query 22 of 44, the current import will be aborted and restarted from beginning
The AXL may never be able to complete the sync, thus new users and devices could never appear in the IAS and, upon expire of the old cache and depending on the configuration, the old users may be changed or deleted at the next user sync cycle
Cause
CUCM rate-limits the AXL queries and the IAS AXL client exceeds said limit
Solution
Increase the limit to 18 under the CUCM Enterprise Parameters for Rate Control → Allowed Device Queries Per Minute web interface
RDP into the IAS server and edit the %IAS_ROOT%\Apps\ApplicationSuite\Component\Native\SvSasAXL\SvSasAXL.ini file, add the following line under the [Settings] section:
EnquiryingTime=<N>
By default, N is 5, try to increase it to 30, ie:
EnquiryingTime=30
And monitor the AXL log file as explained above, if the error persists, increase it by 5 until it does not happen anymore