Cloud & Telecom Security

Entering Internet age, new software vulnerabilities are found every day and the corresponding patches and temporary workarounds are coming with them. The software vulnerabilities may cause system breakdowns or are being exploited at any time. As the mass of installations of the patches costs lots of system resourse or may cause the restarting of the system, and performance decreases. On the other hand, rushing on patch might not be secure or might bring potential dangers to the stability and functionality of a system. How can these problems be solved? The patch and vulnerability management is not only the business of the security administrators but also the focus of the entire IT operation sector. We would like to share our knowledge and best practices on managing patches and vulnerabilities with the industry.

Figures and problems

Lets read some figures first. As reported by Meta Group, a total of 4192 vulnerabilities were found in 2002 and at the same time, as per the real statistic, system administrators cost a total of 1920 work hours to make up all 4 patching to 120 servers. This means that it will take about 4 hours to patch a server – including backup, installation and debugging. Suppose the system administrators are skilled and they can completely learn vulnerability and patch solution within 20 minutes. This needs 172 persons to cost an entire working day for making up the 4192 vulnerabilities. In case of only10% – 413 vulnerabilities – of them are adapted to our own network environment and each correspondent patch is on 10 servers, it needs 2065 persons/day (there are about 10 servers with the same configurations. From these figures we have learnt that by adding up the days of the two persons, there needs to be almost 10 full time administrators, while that doesn’t include the processes of testing and validating the issued patching and the secondary resource consuming resulted from the failure of patching making up. Thus we can see that the patching and vulnerability management has been a huge resource funnel and wastes a lot of system management resources.

The administrators shouldn’t neglect these potential security problems and such huge resource waste is not affordable. More effective solutions must be found, these include:

•  Introduce knowledge sharing or an external knowledge base (special services), reduce the resource consuming when learning vulnerabilities and patches.
•  Establish an effective patch management process and a backup feedback plan.
•  Introduce into a roboticized patch and vulnerability management tool to accelerate the deployment.
• Is patching necessary?

From the above-mentioned figures we see that patching is not free at all, on the contrary, it is costly. The costs of patching include collecting, learning, testing, deploying, backup recovery and risk costs. But the costs of not patching include data loss, theft, falsification, denial of service, system recovering and other intangible losses. In general, when the following cost inequality forms, it is time to patch:

Deploying costs + patching failure rate × failure costs <= intrusion rate × cost of intrusion rate

Three figures are commonly used to describe risk features of a vulnerability: I (Impact – intrusion cost), P (popularity – how many people know the vulnerability), S (simplicity – how difficult to exploit the vulnerability). P×S may reflects the intrusion rate. By this way, I×P×S at the right side of the inequality can say the costs correspondent to patching or not patching. It substantially reflects the defense strength of the patching management decision (acceptable or not). For reference, see the following sketch map.

Diagram: View patching management from the point of view of risk management

At the top of the curve, even the patch with 0 product of IPS (with tiny risks) must be completely deployed as per the requirement of 100% security. Then the cost is extremely high and generally doesn’t comply with the principle of investment yields.

From the above inequality we learn that the deployment cost plus the cost of patching failure decides the cost-effective IPS strength. If we can reduce the deployment cost and the patching failure rate by testing, the losses and cost resulted from patching can be minimized on the premise of winning back the investment yields so as to improve the defense strength that is decided by IPS.

Earlier or later
It is generally believed that patching should be deployed as quickly as possible when it is found. But as per a research figure on CVE vulnerabilities and patches that is issued by USENIX of USA, about 18% patching are redeployed later － re-patch the patch. It means that 18% of patching that is patched at the first response time may cause new defects or security vulnerabilities. Along with time, the security and stability of the patching itself increases while the risks of losses resulted from it decreases correspondently.

Refering to the following diagram, we must balance the failure that might be caused by earlier patching and the intrusion risks and costs that might be caused by later patching so as to select better patching time. As per the real search on CVE vulnerabilities and patches, it will cost 2-4 weeks to stabilize a new issued patch, in other words, two weeks after the patch is issued is the best time to consider deploying it as soon as possible. If such a delay doesn’t satisfy you, you should establish a quicker and stronger security laboratory of specializing in researching the test on patches to shorten the large-scale patch deploying time to be within one week.

Diagram：Proper patching time

Automation:
From the inequality we learn that the reducing deployment and patching failure costs can improve the security defense strength resulted from patch management.

An effective way of reducing deployment cost is automation. Establish an automatic patching knowledge base and management system that can cover the entire net to collect, set up and distribute a patch package. Such automatic system can bring in the following great benefits:

•  Minimize the time window of the entire patch distribution.
•  Lower greatly the work time of each hour/server/patching －and lower distribution & installation costs to near nil while only leaving the work time costs for producing the software distribution package, checking and testing the patch installation.
•  Ensure the consistency of managing patch deployment within the entire network.

In addition, a way of reducing the cost of patching failure is testing them effectively by purchasing the services from the special manufactures or building your own security laborites. This investment can yield marvelous benefits for the large distributed enterprises.

Processes
The patch management cannot be predicted well due to the features of patch. But as an administrator, he/she must predict the meanings and features as well as the problems occurring during operation of patch management for processes and means. All patch distribution and management tools are just for helping the acceleration and automation of correspondent strategies and processes, as well as for improving efficiency and quality, but the logic cannot be changed. If the patch management process is in chaos, the outcome of the automation must be worse than chaos.

The following substantial processes must be taken into consideration when designing relevant patch management strategies and processes:

• Security strategy of an enterprise: As vulnerabilities or defects are threatening security, security strategy should be developed to cover the vulnerabilities and patches. Patch and vulnerability management processes should be adapted to the hereinabove strategies.
• Change and release management: Producing, testing, approving and distribution of patches should be defined in standardized change and release processes. If there are not such completed change and release processes, the priority or classification, and the relevant titles, duties and time of producing, debugging, approving and deploying of patches should be defined in the patch management processes.
• Configuration management: As per the best practice of ITIL, a completed and consistent central configuration management database is the guarantee for keeping at high level IT service management. Define the configuration management items related to patches and vulnerabilities and ensure the timely and accurate renewal through processes.
• Asset management: If there is an asset management system and process, the patch and vulnerability should be correspondent with the asset and relevant business priority so as to produce specified processes of different key assets.
• Backup recovery and business continuity management: As related to patch deployment, an enterprise backup recovery and a business continuity management plan must be fully taken into consideration and they can be changed when necessary.
• Emergency response: A correspondent emergency response process should be designed to attach to all patch management processes. From the above figures we learnt that a certain rate of new vulnerabilities might occur in the officially issued patches and they connect with allocations in the enterprise system, the vulnerabilities must not be neglected. Such recoveries and responses must be ensured in the processes.

The administrators should completely consider every related aspect and refer to practical experiences in IT service management and ITIL or resort on external specialty services to design patch management strategies and operation processes that match the entire IT infrastructure management system.

Before applying the hereinabove automatic patch management system, the real IT environment and various issues about the entire life cycle of patches should be completely investigated and researched so as to design more precise patch management strategies and processes, or at least define the application, priority, production, testing of distribution package, approving and distribution installation, testing after installation and backup recovery plan of the patch management.

Be Sociable, Share!

•