Performance Testing in a Provisioning World

I’m often asked how many devices-per-second we can provision. The answer is far more complex than just saying 2,000 devices per second, for example. You can compare this to the car seller’s adage: “Your mileage may vary.” In provisioning, it definitely will vary, and it can be difficult to say by how much.

First, consider what’s involved in provisioning. We first must find the subscriber in a database. If the database is local (client class), then this lookup is very fast. If the subscriber is a member of multiple client classes, then this introduces a small increase in processing time. If the subscriber resides in LDAP, the delay is more significant. There are other factors to consider too. Where is that data hosted? What kind of hardware is being used? It’s impossible for us to predict ahead of time.

Then comes the time it takes to build the configuration files. The more complex the file, the longer it will take us to build. Other variations can occur on the network itself and the CMTS. By themselves, none of these factors will impact on the time it takes to provision a device. However, when added together, and over tens of thousands of devices, the delays become more significant.

When you consider all these issues, it’s easy to understand that we cannot run every possible variation when we test performance. Otherwise, it would take years to report performance numbers. Instead, we test a standard configuration to gain performance data. But how do we generate 250,000 cable modems? Surely we do not have that many in our lab! While that would be cool, we’ve built a simulator that can generate the full DHCP cycle. Even so, this can’t account for actual DOCSIS delays that may be caused by the HFC network or the registration process. As a result, your mileage will vary.

We use our performance numbers internally to compare between versions to make sure that any new features will not affect performance. So take any vendor promising great riches with a grain of salt. We never claim anything we can’t deliver and try to make our testing as realistic to real world scenarios as possible.