TWR DATA PROCESSING 2003


  •         INTRODUCTION

    The main goal of TWR data processing is to have data filtered at level L1 with the additional waveform information.
    In 2003, both muon DAQ and TWR DAQ ran multiplicity trigger M = 24, so GPS time matching is straightforward.
    Selection of bad runs, bad OMs, etc. was done using information from Muon DAQ. Since we expect the most benefit from High Energy (HE) analysis, we used scripts developed for Muon-DAQ for low level filtering of raw TWR data.

     

  •         METHOD

    To process TWR data there are two possibilities:

    - 1. using a raw TWR reader (but no reader available for 2003 or 2004 TWR raw format as of Dec. 15, 2004).

    - 2. merging the TWR data with Muon data. We opted for the second solution.
    For the merging solution the following software package have been used: ROOT, RDMC, Merger-software. These packages run at UCI (Fedora Core 1 Kernel 2.4.22-1.2199.nptl i686 i686 i386 GNU/Linux) and at SDSC machines (SuSE Linux Kernel 2.4.21.SuSE_241.bef1 ia64).

     

  •         TWR DATA 2003

    - The entire set of TWR data is recorded on 102 SDLT tapes [TAPE_003 - TAPE_104] (available at UCI). Since TWR data from 2003 is more than 15 TB (after compression), it is not available online. However, 20% of merged is now available: TWR L1-FILTERED DATA 2003.

    - All data using temporary hard-drive has been transferred to large data storage system, the Storage Resource Broker (SRB) System at San Diego Supercomputer Center (SDSC)

    - All software (ROOT, RDMC, Merger-software) has been installed successfully on IA64-bit machines at SDSC

    - All necessary scripts to run the processing and merging chain has been developed and are running successfully at SDSC.

    - For merging process two data inputs are necessary: muon DAQ data, TWR DAQ data. For the muon data, the L1-filtered data provided by the Zeuthen group was used. This was downloaded to UCI and transferred to the SRB system at SDSC.

    - The processing chain consists of: convert raw TWR format into ROOT format readable by the merger software, look for GPS data coincidence between L1-filter f2k data and ROOT twr-data, merge both data sets using (TWR-process scripts), write f2k output containing the original muon DAQ data with entire waveform information for single events, and copy to UCI disk.

    - The final output contains all data streams at level L1, like down going muon, cascade, high energy trigger, WIMPs, and other which where selected in the Sieglinde Steering files SLsteering_file2a_v2_online.L1

    - When completed, the entire data volume at L1 of the f2k merged output will be approximately of 3TB and copies of processed and merged data at L1-level will be backed up on SDLT tapes (approx. 15-20 SDLT tapes).
    At current rate of progress, the merging should complete by mid-January 2005.

    - AMANDA collaborators who wish to get more than 20% of the merged data, should contact (Andrea Silvestri). We can either try to buy 4TB hard-drive for downloading via network or copy onto SDLT tapes.

     

  •         PROCESSED TWR DATA 2003

    - Now available: 20% (Feb.-Apr. 2003) of TWR and Muon-DAQ merged data: TWR L1-FILTERED DATA 2003
    (Data are subdivided in directory like TAPE_XXX containing approximately 550 file of 60 MB each)
    - Status of remaining TWR and Muon-DAQ merged data : a summary table will be soon available. Merging is continuing as time goes on.

  •         TWR MC MASS PRODUCTION

    TWR MC mass production will be performed at SDSC. The software currently used to generate MC events with waveforms is the Simuperl package, which runs CORSICA, MMC, MAM, and AMASIM.
    Simuperl has been updated to run AMASIM (magnesium release), which includes the TWR hardware in the detector simulation. Small output files (f2k) with a total of 6000 background events at trigger level are available TWR MC events 2003. These files were generated at UCI (3 hr 1-CPU time for 6000 events) and TWR MC files will be soon generated at SDSC. Currently tests are on-going to check the MC and experimental data comparison, like waveforms, hits distribution, and other important variables crucial for properly reconstructed events.
    After comparison between Muon and TWR data is proven and satisfactory, and AMASIM comparison to TWR is satisfactory, a large mass production will start to generate (considering the rate of generating file and the rate of filtering them) a comparable number of events to the experimental data. Also MC data at trigger level would reach a data volume of about 10TB after compression.

  •         TWR CONFIG FILES 2003 and 2004

    - Here available: The TWR config files provided by Andreas Gross for data 2003 and 2004: TWR CNF files 2003 and 2004
    The format of the TWR.cnf_2004 file differs compared to the TWR.cnf_2003 format due to the VLF veto periods, and the TWR.cnf_2004_reduced_hv file format differs due to the reduced HV applied since end of November 2004.

  •         A SAMPLE OF RAW TWR DATA

    - Here available a sample of raw TWR data 2003, from TAPE_067 (Days 249-252) Raw TWR Data 2003


     

    Andrea Silvestri
    Last modified: Thu Dec 17 15:19:59 PST 2004