Metrics of Interest for 50k Scans Biweekly - Lab Environment

Test Summary

Conclusion

This testing is running against 50,000 devices (per device per scan) in a simulated environment. The time spent on scanning the remote devices are not counted in; the network latency does not exist since the simulated environment is running on the lab in the same VLAN, which means there is no scanner connect failure. The auto-deletion seldom happen because there is no device change in the simulated environment.

The whole testing finished in one week. Considering this result, it is confident that in a biweekly schedule, 50,000 devices can be scanned in a real customer environment, thus Data Flow Probe capacity can be increased to 50,000 devices, with each device contributing 360 CIs and relationships, and the total data volume will be 17.7 million CIs and relationships in UCMDB server.

Background

This testing is designed to simulate a scenario of 50,000 devices by running 3 jobs (Range IPs by ICMP, Host Connection by Shell, and Inventory Discovery by Scanner), in order to measure how long it takes to finish discovering these devices.

The testing environment is populated first with these 50,000 scan files, with all probe data cleared. This is to make sure there is no redundant and touching which will speed up the whole data-in, the operations on server side are mainly update and merge.

Test Environment

Role in Group	CPU	RAM	OS	Hard Drive	Port	SID
Database Server - Oracle	32 cores 2.40Ghz	128GB	Red Hat Enterprise Linux 7.0 64-bit	1.4 TB	1521	sid=pcoe
UCMDB Server 10.30	16 cores 2.40Ghz	128GB	Windows Server 2008 R2 64-bit	1.3 TB
Data Flow Probe 10.30 simulated environment	32 cores 2.40Ghz	128GB	Windows Server 2008 R2 64-bit	1.3 TB

 

 Probe Configuration

Property files on the probe	Setting Details	Comments
UCMDB\DataFlowProbe\ bin\xmlenricher\ WrapperEnricher.conf	wrapper.java.maxmemory=8192 (the first 4 days) wrapper.java.maxmemory=16384 (the last 2.5 days)	Change the XML Enricher configuration to use "Enterprise" deployment
UCMDB\DataFlowProbe\ conf\enricher.properties	max_enricher_thread_number=4 (the first 4 days) max_enricher_thread_number=8 (the last 2.5 days)	Change the XML Enricher configuration to use "Enterprise" deployment
UCMDB\DataFlowProbe\ bin\WrapperEnv.conf	set.GATEWAY_MIN_MEM=2048 set.GATEWAY_MAX_MEM=8192 set.MANAGER_MIN_MEM=2048 set.MANAGER_MAX_MEM=8192	Increase the amount of memory used by JVM, so that it can run more discovery jobs in parallel
UCMDB\DataFlowProbe\conf\postgresql.conf	shared_buffers = 1024MB	Increase the amount of memory that PostgreSQL can use for buffering the data in memory
UCMDB\DataFlowProbe\ conf\DataFlowProbe .properties	appilog.agent.local.services. poolThreads=110 appilog.agent.local.services. defaultMaxJobThreads=60 appilog.agent.probe.maxConnection =120	Increase the number of threads to be used by the probe. This increases the concurrency of operations and allows more discovery jobs to run simultaneously.

 

UCMDB Server Configuration

Impacting Factors	Category	Default Value	Value for a large deployment	Comment
Max number of Objects in Server	Global Setting Name	20,000,000	30,000,000	These settings are increased to prevent errors that occur once certain fuse types are reaching their default limits
Maximum number of elements (nodes) in view result	Global Setting Name	100,000	4,000,000

 

Job configuration and schedule

Job Name	Schedule	# of Threads
Range IPs by ICMP	daily	1
Host Connection by Shell	daily	60
Inventory Discovery by Scanner	Bi-weekly	40

Test Results

XML Enricher speed:

With wrapper.java.maxmemory=8192 and max_enricher_thread_number=4, it processed about ~6700 scan files per day.

With wrapper.java.maxmemory=16384 and max_enricher_thread_number=8, it processed about ~10000 scan files per day.

The full discovery cycle:

7 days for 43372 devices.

Data-in CI Rate:

~96 CIs/second for Host Connection by Shell

~300 CIs/second for Inventory Discovery by Scanner

Conclusion:

The test confirms that 50K devices per probe is possible.

However, the speed of processing the scan files in your environment may vary depending on the CPU type, disk speed, and crucially scan file size.

Impact Factors for the Capacity - 50K

The data in the table below are collected during the testing for the probe capacity of 50000/360 scan per week, ignoring the time scanner connects and scanner runs in the target node to generate the scan file.

Impact Factors show what will impact the probe capacity. The ones listed here are very important ones.

The Value column shows the value used for the impact factor during our testing.

Impact Factors	Value used in testing	Comments
Number of devices in the environment	50,000
Average number of related CIs per scan	380	Operation system types impact the number of installed software CI a lot. For example, enabling BaseUnixOs.zsai for the XML Enricher to process UNIX scan files will introduce a lot of installed software CIs.
Frequency of new IP address assignment by DHCP	50,000 data center devices: Fixed IPs
The number of management zones used	0	Refer to the management zone configuration.
Connect failure ratio	0
Network latency between probe & discovery devices	In the same LAN: < 5ms	Usually this is the ping time from the Data Flow Probe to the nodes being discovered
Number of IPs	50,000
Devices online timeslots	Data Center: 24 hours/day
Change Ratio for the discovered CIs	0	The higher change ratio of discovered CIs, the longer time it takes to push the changed CIs to the UCMDB server
Discovery jobs	For fixed IPs: Range IPs by ICMP Host Connection by Shell Inventory Discovery by Scanner
Total number of discovery threads per probe	110
# Threads of Host Connection by Shell	60
# Threads of Inventory Discovery by Scanner	40
Schedule of Inventory Discovery by Scanner	Bi-Weekly: 24x7, no blackout policy
Number of total CIs in UCMDB	17.7 million (CIs & relationships)	InstalledSoftware and UserSoftwareUtilization CIs (if software utilization is enabled) usually contribute the largest share of CIs/relationships.
The size of probe DB tables	ddm_discovery_results: 8 million ddm_discovery_touch_results: 8 million ddm_map_objectid: 8 million	When the size of DB tables reach 8 million, the operation of getting Data Flow Probe Status from UI may encounter timeout issue sometimes. From the testing, the timeout ratio is 50%. When increasing to 10 million, getting Data Flow Probe Status from UI operation always encounters timeout issue.