delays in /registry/request/copy

[vlimant]

making calls to that api is taking 20 [s] to 150 [s] as timed here
CMSCompOps/WmAgentScripts@2ba82c5

[vlimant]

with CMSCompOps/WmAgentScripts#353 there is going to be more concurrent calls to that API

[vlimant]

calls are now failing with

[s] to hand over to dynamo of [u'/ADDMonoZ_MD_2_d_2/RunIIAutumn18MiniAOD-102X_upgrade2018_realistic_v15-v1/MINIAODSIM']
Failed _pass_to_dynamo

<title>500 - Internal Server Error</title>

500 - Internal Server Error

[dr-stringfellow]

This is where it is taking very long - proportional to the number of sites.

https://github.com/SmartDataProjects/dynamo/blob/master/lib/request/copy.py#L33

Further debugging needed.

[dr-stringfellow]

https://github.com/SmartDataProjects/dynamo/blob/master/lib/request/common.py#L136

This is the critical step.

@yiiyama , do you have any idea?

[vlimant]

is this a limitation in the database schema, or in how the constraints are made?

may I ask : why not using py-sql packages like sqlalchemy and simplify db business?

[yiiyama]

That's a particularly nasty query, indeed. Adding indices to site in https://github.com/SmartDataProjects/dynamo/blob/master/mysql/schema/dynamoregister/copy_request_sites.sql and item in https://github.com/SmartDataProjects/dynamo/blob/master/mysql/schema/dynamoregister/copy_request_items.sql might help. In general, there may be a more efficient way to do the same thing though.

@vlimant There's no reason. I just didn't invest time initially to look around for py-sql solutions and so didn't know about sqlalchemy until fairly far into the dynamo development.

[vlimant]

how can we get this to run in <2s instead of >30s ?
the server keeps on crashing with

Sending /RSGravitonToGammaGamma_kMpl01_M_1250_TuneCP2_13TeV_pythia8/RunIIFall17NanoAODv4-PU2017_12Apr2018_Nano14Dec2018_102X_mc2017_realistic_v6-v1/NANOAODSIM to DDM
{'item': [u'/RSGravitonToGammaGamma_kMpl01_M_1250_TuneCP2_13TeV_pythia8/RunIIFall17NanoAODv4-PU2017_12Apr2018_Nano14Dec2018_102X_mc2017_realistic_v6-v1/NANOAODSIM'], 'site': ['T2_*
', 'T1_*_Disk'], 'n': 1}
Failed _pass_to_dynamo

<title>500 - Internal Server Error</title>

500 - Internal Server Error

[dr-stringfellow]

I don’t understand why this is a problem now. Has something changed in the communication on the unified site? Do we know if it took so long two weeks ago? I didn’t hear of any problems back then. This makes me think something happened to the server/SSD disk and we feel it when we operate with these tables. How long has this been a problem?

[vlimant]

the crash and the delay : I only see them now that I have my eyes on it, while commissioning a multithread application for unified. I have no feel for how often this happened in the past. it surely was not preventing massively things to be announced.
ATM, it looks like multithreading is making this worse by having up to 10 concurrent /copy calls

even single threaded do I see crashes and delays, and this prevents the backlog of things to be announced to be absorbed quickly. It takes 30 min to announce 7 workflows, and with hundreds of those in the system ATM, this is a bottleneck.

let's please find out and fix this

[vlimant]

is there a way to "cache" what is making "T2_, T1_" slow and reuse it over and over in subsequent call within says 30s ?

[vlimant]

I'd like to pull in the changes on unified to accelerate the closing cycle and not have to bother again ; the rate of "500 internal error" seems related to sending requests at higher rate. please fix the API as you can.

[vlimant]

@dr-stringfellow please update with your proposal of the /copy_cache api, and ping here when you are ready so that we put CMSCompOps/WmAgentScripts#357 in

[dr-stringfellow]

Will have an additional table in dynamoregister database where things from unified get cached and dynamo acts on them whenever there is time. Might have to create an extra database ("dynamocache") to prevent locking out unified during the acting on the cached info.