I have a customer that is using the Connector Solution (in 6.0 and 7.x) to import users, group and parameters into their CMDB's for various business reasons.
Over time their import data has grown and so has the process time, to some extremes: updating 10~20K entries on a dataclass from a CSV file containing 1,000,000+ lines (a mere 30MiB) would take 3 hours + (in 6.0, and many more in 7.x).
Note that the imported data is not one for one - i.e. we are not populating a data table but linking keys from other tables, which is different process to handle from the SMP.
In a couple of cases we decided to take the processing outside of the Connector Solution, via a simple SQL procedure. This worked great but it requires some serious amount of work to implement the data insert, update and delete part of the procedure. This allowed us to run the import in less than one minute (fyi, it's Import #3 in the table below).
Thankfully my customer reported (today) an alternative option to this tedious process (and here it is):
Add an index on the Item.Name field!
Yes, it sound so little, but the customer reported the following data which is significant:
| Before | After | |
| Import #1 | 150+ minutes | 26 minutes |
| Import #2 | 120+ minutes | 25 minutes |
| Import #3 | 150+ minutes | 40 minutes |