Searching the Help
To search for information in the Help, type a word or phrase in the Search box. When you enter a group of words, OR is inferred. You can use Boolean operators to refine your search.
Results returned are case insensitive. However, results ranking takes case into account and assigns higher scores to case matches. Therefore, a search for "cats" followed by a search for "Cats" would return the same number of Help topics, but the order in which the topics are listed would be different.
Search for | Example | Results |
---|---|---|
A single word | cat
|
Topics that contain the word "cat". You will also find its grammatical variations, such as "cats". |
A phrase. You can specify that the search results contain a specific phrase. |
"cat food" (quotation marks) |
Topics that contain the literal phrase "cat food" and all its grammatical variations. Without the quotation marks, the query is equivalent to specifying an OR operator, which finds topics with one of the individual words instead of the phrase. |
Search for | Operator | Example |
---|---|---|
Two or more words in the same topic |
|
|
Either word in a topic |
|
|
Topics that do not contain a specific word or phrase |
|
|
Topics that contain one string and do not contain another | ^ (caret) |
cat ^ mouse
|
A combination of search types | ( ) parentheses |
|
Metrics of Interest for 50k Scans Biweekly - Lab Environment
Conclusion
This testing is running against 50,000 devices (per device per scan) in a simulated environment. The time spent on scanning the remote devices are not counted in; the network latency does not exist since the simulated environment is running on the lab in the same VLAN, which means there is no scanner connect failure. The auto-deletion seldom happen because there is no device change in the simulated environment.
The whole testing finished in one week. Considering this result, it is confident that in a biweekly schedule, 50,000 devices can be scanned in a real customer environment, thus Data Flow Probe capacity can be increased to 50,000 devices, with each device contributing 360 CIs and relationships, and the total data volume will be 17.7 million CIs and relationships in UCMDB server.
Background
This testing is designed to simulate a scenario of 50,000 devices by running 3 jobs (Range IPs by ICMP, Host Connection by Shell, and Inventory Discovery by Scanner), in order to measure how long it takes to finish discovering these devices.
The testing environment is populated first with these 50,000 scan files, with all probe data cleared. This is to make sure there is no redundant and touching which will speed up the whole data-in, the operations on server side are mainly update and merge.
Role in Group | CPU | RAM | OS | Hard Drive | Port | SID |
---|---|---|---|---|---|---|
Database Server - Oracle |
32 cores 2.40Ghz |
128GB | Red Hat Enterprise Linux 7.0 64-bit | 1.4 TB | 1521 | sid=pcoe |
UCMDB Server 10.30 | 16 cores 2.40Ghz | 128GB | Windows Server 2008 R2 64-bit | 1.3 TB | ||
Data Flow Probe 10.30 simulated environment | 32 cores 2.40Ghz | 128GB | Windows Server 2008 R2 64-bit | 1.3 TB |
Property files on the probe | Setting Details | Comments |
---|---|---|
UCMDB\DataFlowProbe\ bin\xmlenricher\ WrapperEnricher.conf |
wrapper.java.maxmemory=8192 wrapper.java.maxmemory=16384 |
Change the XML Enricher configuration to use "Enterprise" deployment |
UCMDB\DataFlowProbe\ conf\enricher.properties |
max_enricher_thread_number=4 max_enricher_thread_number=8 |
Change the XML Enricher configuration to use "Enterprise" deployment |
UCMDB\DataFlowProbe\ bin\WrapperEnv.conf |
set.GATEWAY_MIN_MEM=2048 set.GATEWAY_MAX_MEM=8192 set.MANAGER_MIN_MEM=2048 set.MANAGER_MAX_MEM=8192 |
Increase the amount of memory used by JVM, so that it can run more discovery jobs in parallel |
UCMDB\DataFlowProbe\conf\postgresql.conf | shared_buffers = 1024MB | Increase the amount of memory that PostgreSQL can use for buffering the data in memory |
UCMDB\DataFlowProbe\ conf\DataFlowProbe .properties |
appilog.agent.local.services. poolThreads=110 appilog.agent.local.services. defaultMaxJobThreads=60 appilog.agent.probe.maxConnection =120 |
Increase the number of threads to be used by the probe. This increases the concurrency of operations and allows more discovery jobs to run simultaneously. |
Impacting Factors | Category | Default Value | Value for a large deployment | Comment |
---|---|---|---|---|
Max number of Objects in Server | Global Setting Name |
20,000,000 |
30,000,000 | These settings are increased to prevent errors that occur once certain fuse types are reaching their default limits |
Maximum number of elements (nodes) in view result | Global Setting Name | 100,000 | 4,000,000 |
Job Name | Schedule | # of Threads |
---|---|---|
Range IPs by ICMP | daily | 1 |
Host Connection by Shell | daily | 60 |
Inventory Discovery by Scanner | Bi-weekly | 40 |
XML Enricher speed:
With wrapper.java.maxmemory=8192 and max_enricher_thread_number=4, it processed about ~6700 scan files per day.
With wrapper.java.maxmemory=16384 and max_enricher_thread_number=8, it processed about ~10000 scan files per day.
The full discovery cycle:
7 days for 43372 devices.
Data-in CI Rate:
~96 CIs/second for Host Connection by Shell
~300 CIs/second for Inventory Discovery by Scanner
Conclusion:
The test confirms that 50K devices per probe is possible.
However, the speed of processing the scan files in your environment may vary depending on the CPU type, disk speed, and crucially scan file size.
The data in the table below are collected during the testing for the probe capacity of 50000/360 scan per week, ignoring the time scanner connects and scanner runs in the target node to generate the scan file.
Impact Factors show what will impact the probe capacity. The ones listed here are very important ones.
The Value column shows the value used for the impact factor during our testing.
Impact Factors | Value used in testing | Comments |
---|---|---|
Number of devices in the environment | 50,000 | |
Average number of related CIs per scan | 380 | Operation system types impact the number of installed software CI a lot. For example, enabling BaseUnixOs.zsai for the XML Enricher to process UNIX scan files will introduce a lot of installed software CIs. |
Frequency of new IP address assignment by DHCP | 50,000 data center devices: Fixed IPs | |
The number of management zones used | 0 | Refer to the management zone configuration. |
Connect failure ratio | 0 | |
Network latency between probe & discovery devices | In the same LAN: < 5ms | Usually this is the ping time from the Data Flow Probe to the nodes being discovered |
Number of IPs | 50,000 | |
Devices online timeslots |
Data Center: 24 hours/day |
|
Change Ratio for the discovered CIs | 0 | The higher change ratio of discovered CIs, the longer time it takes to push the changed CIs to the UCMDB server |
Discovery jobs |
For fixed IPs:
|
|
Total number of discovery threads per probe | 110 | |
# Threads of Host Connection by Shell | 60 | |
# Threads of Inventory Discovery by Scanner | 40 | |
Schedule of Inventory Discovery by Scanner | Bi-Weekly: 24x7, no blackout policy | |
Number of total CIs in UCMDB |
17.7 million (CIs & relationships) |
InstalledSoftware and UserSoftwareUtilization CIs (if software utilization is enabled) usually contribute the largest share of CIs/relationships. |
The size of probe DB tables |
ddm_discovery_results: ddm_discovery_touch_results:
ddm_map_objectid: |
When the size of DB tables reach 8 million, the operation of getting Data Flow Probe Status from UI may encounter timeout issue sometimes. From the testing, the timeout ratio is 50%. When increasing to 10 million, getting Data Flow Probe Status from UI operation always encounters timeout issue. |
We welcome your comments!
To open the configured email client on this computer, open an email window.
Otherwise, copy the information below to a web mail client, and send this email to cms-doc@microfocus.com.
Help Topic ID:
Product:
Topic Title:
Feedback: