CMS: Release of Run2017E and Run2024F RAW data

[tpmccauley]

This issue is for tracking the work needed and the progress towards release of the approved CMS RAW datasets: 1M events of EphemeralHLTPhysics RAW data, collected during run 383631 on July 25th 2024 (Run2024F era).

/EphemeralHLTPhysics0/Run2024F-v1/RAW#35e5ed22-a9dd-4734-833c-70d30de920d7
/EphemeralHLTPhysics1/Run2024F-v1/RAW#9c9e0c57-56a1-4ee4-98dc-384c94184bc9
/EphemeralHLTPhysics2/Run2024F-v1/RAW#91741d2b-3627-4122-822c-edb9b55a6b0c
/EphemeralHLTPhysics3/Run2024F-v1/RAW#3893e7b4-4296-4404-8f21-6264ddc37a05
/EphemeralHLTPhysics4/Run2024F-v1/RAW#a423ec4f-23db-4bee-9562-08699424f779
/EphemeralHLTPhysics5/Run2024F-v1/RAW#f0c884ab-bd0b-4514-a93b-bdaf29b012ec
/EphemeralHLTPhysics6/Run2024F-v1/RAW#848378da-3b39-4a1e-879d-722172fbd356
/EphemeralHLTPhysics7/Run2024F-v1/RAW#11214908-2511-4006-9b27-a040b975e663

• Data files on EOS
• Make container for software environment (which CMSSW?, conditions db?, ...)
• Make record json files (do we need a new repo/code in data-curation)?

[tpmccauley]

There will also be a format derived from the RAW.
See /eos/cms/store/group/tsg/OpenData/store/data/Run2024F/EphemeralHLTPhysics/FED/v1/

We should perhaps have a second record for these derived files and link the two records

[jmhogan]

Using this issue to track both the 2024 RAW and the CICADA RAW request from 2017E.

I'll come and update these rules with folder names within /eos/opendata/cms/upload, but it will probably be self-evident from timestamps.

Rules for CICADA, all for storage on T3_CH_CERN_OpenData_Stage:

• Data: bc300ce88aed49eab23f5f67af946bb8
• QCD 1: db72b7796ec4450aaafbc68e8edc4be4
• QCD 2: d0cebcb5f6804bdca614f2c1dab8f662
• Neutrino 1: 42c73d77c1e3482a82642cb0c4da6c13
• Neutrino 2: f30a031aa7544c7faca967f4d33726e4
• ttbar 1: 8f412381486a4f7d949ccfc65f72cf53
• ttbar 2: 0ad918ed145f434aa063f7751aae1a7f
• ttbar 3: 10f628941fb349929f667ead3440a969

Rules for the 2024 RAW for TSG, also to T3_CH_CERN_OpenData_Stage:

• EphemeralHLTPhysics0: 35c9003a180644a9b54645bd937951db
• EphemeralHLTPhysics1: ddc1195f4153428d8f6f24cfdbf63cf7
• EphemeralHLTPhysics2: 64518514af934385b50bffc4e9a36cd0
• EphemeralHLTPhysics3: e77ec0db58ea41c38ad1a9a67d77a405
• EphemeralHLTPhysics4: fb4d1e3415ae41d3bf00936c57a17384
• EphemeralHLTPhysics5: 82c03dae2570463dac07fcad8ac78d64
• EphemeralHLTPhysics6: f6353a4bd9e54b14a2fad4ece2bd87c1
• EphemeralHLTPhysics7: 1105c983756c4be689592dfe00b485d1

For the 2024F I am also starting an xrdcp of the processed .raw files into /eos/opendata/cms/upload/Run2024F/EphemeralHLTPhysics/FED/v1/run383631/

(accidentally closed the issue, fixed 😆 )

[jmhogan]

@tiborsimko @psaiz:

The derived data .raw files are ready to move. I'm thinking of a path like this:

/eos/opendata/cms/upload/Run2024F/EphemeralHLTPhysics/FED/v1/run383631/*.raw --> /eos/opendata/cms/derived-data/FED/XX-May-25/EphemeralHLTPhysics/run383631/*.raw

The Rucio transfers are almost done, a few files are still transfering

[jmhogan]

ok @tiborsimko @psaiz all the rules have transferred completely and can be moved out of upload

[psaiz]

Thanks a lot! All the rules have been replicated to T3_CH_CERN_OpenData

[tpmccauley]

Adding more information on the 2017E RAW and MC:

/ZeroBias/Run2017E-v1/RAW#37d3f45e-a4a5-11e7-bbce-02163e01ab31 (run numbers 304125 and 304144)

/QCD_Pt-20toInf_MuEnrichedPt5_TuneCP5_13TeV_pythia8/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#8a21edce-3fe1-4333-8c1c-d170a82e7a7d

/QCD_Pt-20toInf_MuEnrichedPt5_TuneCP5_13TeV_pythia8/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#963c1549-f01d-4c10-a08e-7ded7ae67200

/SingleNeutrino/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#148df7d0-6d8b-4701-bee4-c03527e35e02

/SingleNeutrino/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#5761aa1b-6695-4b26-9e2a-144ba6ff7d95

/TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#705a74db-5d12-4bdf-b94c-ce671feb9469

/TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#39caf4ed-bfd1-4f67-bd6e-d54858a84cd3

/TTToHadronic_TuneCP5_13TeV-powheg-pythia8/RunIISummer20UL17HLT-PUForTRKv2_TRKv2_94X_mc2017_realistic_v15-v2/GEN-SIM-DIGI-RAW#83bb42c3-0ceb-4c19-b4fe-f957ecb2df26

[tpmccauley]

to-do for Run2017E samples:

• transfer data to EOS (see /eos/opendata/cms/Run2017E/ZeroBias/RAW/v1/)
• transfer MC to EOS (see /eos/opendata/cms/mc/RunIISummer20UL17HLT/)
• make data records
• make MC records
• derived records
• container?

[jmhogan]

@tpmccauley @psaiz One more todo:

@tiborsimko @psaiz:

The derived data .raw files are ready to move. I'm thinking of a path like this:

/eos/opendata/cms/upload/Run2024F/EphemeralHLTPhysics/FED/v1/run383631/*.raw --> /eos/opendata/cms/derived-data/FED/XX-May-25/EphemeralHLTPhysics/run383631/*.raw

[tpmccauley]

@jmhogan The files have already been moved and it looks like it maps to the path in upload:

EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/> ls
EphemeralHLTPhysics0
EphemeralHLTPhysics1
EphemeralHLTPhysics2
EphemeralHLTPhysics3
EphemeralHLTPhysics4
EphemeralHLTPhysics5
EphemeralHLTPhysics6
EphemeralHLTPhysics7
EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/> cd EphemeralHLTPhysics0/
EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/EphemeralHLTPhysics0/> ls
RAW
EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/EphemeralHLTPhysics0/> cd RAW/
EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/EphemeralHLTPhysics0/RAW/> ls
v1
EOS Console [root://eospublic.cern.ch] |/eos/opendata/cms/Run2024F/EphemeralHLTPhysics0/RAW/>