A number of customer have been reporting a number of increased connections to their EMC Centera Devices.
I thought I’d investigate a little further, I wrote a test app using the EMC Centera SDK, spun up 10 threads using the same poolref
C:\tools>RetrieveContent.exe Enter the IP address or DNS name of the cluster(s): 10.14.96.21,10.14.96.22,10.14.96.23,10.14.96.24 [10.14.96.21] Opened Pool, using poolref: 348777130463956 Press ENTER to continue. Opened Clip:BLGE3TDLPU5FPe2VCP2NS8BOHV3G4185NGU78J0H2I8N0JK09FF2S Opened Clip:16995546G1S78e0J6C74GH4F9D5G4185NH3TM609CDUCRNH56LDFP Opened Clip:9CJLJ21FFI1G5eA9HDEQ07JENT5G4185NML7760DB1V1KQ5OMCKL9 Opened Clip:CUVD1C17HQHI7e1LF2HJITQT4DEG418FA7MIAA0R4GRLR7MQEEFK8 Opened Clip:0SDHQGQ2C5U26eAQ9G4R6JURLJRG418FENIELL0R5AFE52NKMJN3Q Opened Clip:BP4CJJ66S6794e34PHJ2GFURC0AG418FENKNNE0P0ALMKSKHE08J6 Opened Clip:47BSSCJKJ0VLFeFVL4RP4TN49U0G418FEOEQLQ0R1V5RIBSIFO8TA Opened Clip:4PGHG674NIEK0e9298KBEPIM86HG418FEOH3NH0G4EBQIKSDN8U0C Opened Clip:8A8LF6D0C32LGeFSNHNLIR2T76LG418FEOL7GP0LF2UK5PAJ4QLPB Opened Clip:FN5IOVTVGEPJ1eECMM7VU7ND3OKG418FEOT0LS0PDDHGFC9JB0IH6 and pausing.... Press ENTER to continue
I have 4 Access Nodes (AN), and the SDK looks like it opens one connection per AN:
I noticed I had quite a few StorageFileWatch connections, so using a Dtrace filter of poolref:
25,659 14:53:56.843 [5,692] (StorageFileWatch) <7912> EV:M CPools::Open (Increment) -- Connection string: 10.14.96.21, PoolRef: 342777130463912, Usage count: 10 25,670 14:53:56.865 [5,692] (StorageFileWatch) <7912> EV:M CPools::Open (Increment) -- Connection string: 10.14.97.61,10.14.97.62,10.14.97.63,10.14.97.60, PoolRef: 350233193691504, Us 26,009 14:53:56.911 [5,692] (StorageFileWatch) <7492> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,027 14:53:56.935 [5,692] (StorageFileWatch) <7336> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,059 14:53:56.951 [5,692] (StorageFileWatch) <5732> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,080 14:53:56.976 [5,692] (StorageFileWatch) <7904> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,098 14:53:57.011 [5,692] (StorageFileWatch) <7744> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,116 14:53:57.035 [5,692] (StorageFileWatch) <5424> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,288 14:53:57.067 [5,692] (StorageFileWatch) <7912> EV:M CPools::Close (Decrement) -- Connection string: 10.14.96.21, PoolRef: 342777130463912, Usage count: 9 26,293 14:53:57.082 [5,692] (StorageFileWatch) <7912> EV:M CPools::Close -- Enable normal timeout -- Connection string: 10.14.97.61,10.14.97.62,10.14.97.63,10.14.97.60, PoolRef: 35023 26,466 14:54:56.868 [5,692] (StorageFileWatch) <5632> EV:M CPools::Open (Increment) -- Connection string: 10.14.96.21, PoolRef: 342777130463912, Usage count: 10 26,477 14:54:56.889 [5,692] (StorageFileWatch) <5632> EV:M CPools::Open (Increment) -- Connection string: 10.14.97.61,10.14.97.62,10.14.97.63,10.14.97.60, PoolRef: 350233193691504, Us 26,809 14:54:56.925 [5,692] (StorageFileWatch) <7492> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,827 14:54:56.949 [5,692] (StorageFileWatch) <7264> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,845 14:54:56.974 [5,692] (StorageFileWatch) <5168> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,884 14:54:57.000 [5,692] (StorageFileWatch) <4680> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,902 14:54:57.023 [5,692] (StorageFileWatch) <7336> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 26,926 14:54:57.047 [5,692] (StorageFileWatch) <6136> EV:M CVaultStoreEMCCentera::PoolOpen - Using existing PoolRef: 342777130463912 27,095 14:54:57.082 [5,692] (StorageFileWatch) <5632> EV:M CPools::Close (Decrement) -- Connection string: 10.14.96.21, PoolRef: 342777130463912, Usage count: 9 27,100 14:54:57.096 [5,692] (StorageFileWatch) <5632> EV:M CPools::Close -- Enable normal timeout -- Connection string: 10.14.97.61,10.14.97.62,10.14.97.63,10.14.97.60, PoolRef: 35023
I see two poolref’s as per my connection list:
select PartitionName,IPAddressList from PartitionEntry
ggg Ptn4 10.14.96.21,10.14.97.61
ggg Ptn9 10.14.96.21
So in summary an EV Server can have multiple processes (StorageCrawler, StorageFileWatch, StorageArchive, etc.) that connect to a Centera. If all connections within a single process use the same connection string we will open one PoolRef. Therefore multiple threads within a single process will share the same PoolRef. PoolRef sharing can only occur within a single process.
In EV10, a number (1+) of StorageCrawler processes (default of max 10) will be maintained on each Storage server to handle indexing requests. Processes are only launched on a ‘need to’ basis so it is possible that none will be running if no work is being requested from that particular Storage Server. This model will boost StorageCrawler’s ability to cope with 64-bit demand from multiple Indexing servers and reduces the existing single point of failure.