Once the above steps have been taken, one can procede to catalogue data in the database. This can be done through either the FORTRAN callable interface or the FATMEN shell. We will demonstrate the use of both and show how the FATMEN system can then be used to access the data referenced by these catalogue entries.
The following examples show how existing data may be catalogued within FATMEN. The examples are taken from the Fermilab CDF experiment.
Information on existing data is kept in flat files, as shown below.
R015881E.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:06:53:19.60|GOOD| R015882B.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:08:59:03.96|GOOD| R015882C.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:10:25:58.47|GOOD| R015882D.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:11:50:21.63|GOOD| R015898A.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:13:26:08.62|GOOD| R015910A.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:14:13:17.86|GOOD| R015911A.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:15:43:56.84|GOOD|
where the fields are dataset name (filename), tape number (vsn), nodename, device name on which the tape was written, date and time and status.
The easiest way to add this information is to write a small command file that converts the file into a KUMAC file. This can be done as follows.
$ open/read in top04.tape $ open/write out addfat.kumac $loop: $ read/end=eof in line $ line=f$edit(line,"TRIM,COMPRESS,COLLAPSE") $ dsn =f$element(0,"|",line) $ vsn =f$element(1,"|",line) $ node=f$element(2,"|",line) $ comm=f$element(5,"|",line) $ fname=f$element(0,".",dsn) $ temp =dsn - fname - "." $ gname="//fnal/cdf/dst/" + temp + "/" + fname $ write out "ADD/TAPE ''vsn' ''vsn' 1 ''gname' ''dsn' _" $ write out "YBB 0 ''node' ''comm' F 0 0 200 3" $ goto loop $eof: close in $ close out
This generates the following KUMAC file.
ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015880A R015880A.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CEH060 CEH060 1 //fnal/cdf/dst/TOP04/R015881B R015881B.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015881C R015881C.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015881D R015881D.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015881E R015881E.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015882B R015882B.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015882C R015882C.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015882D R015882D.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015898A R015898A.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015910A R015910A.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3 ADD/TAPE CED984 CED984 1 //fnal/cdf/dst/TOP04/R015911A R015911A.TOP04 _ UN 0 FNALH GOOD F 0 0 200 3
To add this to FATMEN we now type
EXEC ADDFAT
from within the FATMEN shell.
We could also do the same thing using the FORTRAN interface. This would have the advantage that we could also convert the date and time of creation from VAX format and add that to the catalogue, as is shown in the following FORTRAN program.
The ADDTEST fortran program
PROGRAM ADDTEST
*
* Add stuff to FATMEN catalogue, using CDF tape log files
* A similar function can be performed by using ADDFAT.COM,
* followed by ADDFAT.KUMAC in the FM shell.
*
CHARACTER*256 GENAME,DSN,CHLINE
CHARACTER*80 COMM
CHARACTER*8 HOST
CHARACTER*23 VAXDAT
CHARACTER*6 VSN
CHARACTER*4 FFORM,RECFM,CHOPT
CHARACTER*3 MONTHS(12),CHMON
DATA MONTHS( 1)/'JAN'/,MONTHS( 2)/'FEB'/,
+ MONTHS( 3)/'MAR'/,MONTHS( 4)/'APR'/,
+ MONTHS( 5)/'MAY'/,MONTHS( 6)/'JUN'/,
+ MONTHS( 7)/'JUL'/,MONTHS( 8)/'AUG'/,
+ MONTHS( 9)/'SEP'/,MONTHS(10)/'OCT'/,
+ MONTHS(11)/'NOV'/,MONTHS(12)/'DEC'/
*
* Start of FATMEN sequence FATPARA
*
** *** Data set bank mnemonics
*
*
* Keys
PARAMETER ( MKSRFA= 1, MKFNFA= 2, MKCLFA=7, MKMTFA=8
1 ,MKLCFA= 9, MKNBFA=10, NKDSFA=10 )
*
** *** Bank offsets
*
PARAMETER ( MFQNFA= 1, MHSNFA= 65, MCPLFA= 67, MMTPFA= 68
1 ,MLOCFA= 69, MHSTFA= 70, MHOSFA= 74
2 ,MVSNFA= 77, MVIDFA= 79, MVIPFA= 81, MDENFA= 82
3 ,MVSQFA= 83, MFSQFA= 84, MSRDFA= 85, MERDFA= 86
4 ,MSBLFA= 87, MEBLFA= 88, MRFMFA= 89, MRLNFA= 90
5 ,MBLNFA= 91, MFLFFA= 92, MFUTFA= 93, MCRTFA= 94
6 ,MCTTFA= 95, MLATFA= 96, MCURFA= 97, MCIDFA= 99
7 ,MCNIFA=101, MCJIFA=103, MFPRFA=105, MSYWFA=106
8 ,MUSWFA=116, MUCMFA=126, NWDSFA=145
9 ,MFSZFA=MSYWFA,MUSCFA=MSYWFA+1)
*
* End of FATMEN sequence FATPARA
*
COMMON /USRLNK/LUSRK1,LUSRBK,LADDBK,LUSRLS
DIMENSION NFAT(NWDSFA)
*
* Initialise FATMEN & Zebra...
*
CALL FMSTRT(1,2,'//FNAL/CDF',IRC)
CALL FMLOGL(0)
*
* Open the data file...
*
OPEN(3,FORM='FORMATTED',STATUS='OLD')
*
* Now process the data...
*
10 CONTINUE
READ(3,'(A)',END=99) CHLINE
* PRINT *,'Processing ',CHLINE(1:LENOCC(CHLINE))
*123456789_123456789_123456789_123456789_123456789_123456789_123456789_
*R015880A.TOP04 |CED984|FNALH |MUA21|04-SEP-1989:01:31:48.11|GOOD
*
* Convert date and time...
*
VAXDAT = CHLINE(41:60)
READ(VAXDAT,'(I2,1X,A3,1X,I4,1X,I2,1X,I2)')
+ IDAY,CHMON,IYEA,IHOU,IMIN
IMON = ICNTH(CHMON,MONTHS,12)
IYEA = MOD(IYEA,1900)
ID = IYEA*10000 + IMON*100 + IDAY
IT = IHOU*100 + IMIN
*
* and pack for insertion into FATMEN bank...
*
CALL FMPKTM(ID,IT,IP,IRC)
GENAME = '//FNAL/CDF/DST/'//CHLINE(10:14)//'/'//CHLINE(1:8)
VSN = CHLINE(19:24)
DSN = CHLINE(1:14)
HOST = CHLINE(26:31)
IFILE = 1
CHOPT = 'N'
*
* Change later to YBB...
*
FFORM = 'UN'
ICOPY = 0
RECFM = 'V'
COMM = CHLINE(65:68) // ' ' // CHLINE(35:39)
CALL FMADDT(GENAME,VSN,VSN,IFILE,
+ DSN,FFORM,ICOPY,HOST,RECFM,0,0,0,0,COMM,
+ IVECT,CHOPT,IRC)
*
* Now get back the bank into a vector and modify the fields
* that we could not via FMADDT
*
CALL FMPEEK(GENAME,NFAT,' ',IRC)
*
* Media type 3 = 3420
*
NFAT(MMTPFA) = 3
*
* these should be zero anyway...
*
CALL VZERO(NFAT(MUSWFA),10)
*
* Creation date...
*
NFAT(MCRTFA) = IP
CALL FMPOKE(GENAME,NFAT,'P',IRC)
GOTO 10
99 CLOSE(3)
*
CALL FMEND(IRC)
PRINT *,'Return code ',IRC,' from FMEND'
*
END
Let us assume that the tapes have volume serial numbers (VSN - the magnetically recorded label) JS1001 to JS1010 inclusive. The visual identifiers (VID) are different - CIN136 to CIN145 inclusive. As this type of VID implies, these are actually IBM 3480 cartridges, rather than the older open reel tape. The file identifier of all the datasets is the same - RAWDATA. In order to catalogue these files we must first establish a table linking each file or tape with a generic-name. Let us assume that the generic-name starts with //CERN/OPAL/LEPD/RAWD/P51989. This implies that the data are real LEP data (as opposed to simulated data), in raw data format, i.e. no filtering or reconstruction. The data were recorded in period 5 in 1989.
The FATMEN shell does not allow all fields of the file catalogue to be entered. We are able to specify the following fields:
Suppose we now enter this information into a disk file, each line representing a different tape. The resultant file might look like the following:
JS1001 CIN136 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1002 CIN137 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1003 CIN138 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1004 CIN139 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1005 CIN140 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1006 CIN141 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1007 CIN142 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1008 CIN143 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1009 CIN144 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out' JS1010 CIN145 1 RAWDATA FX 0 VXOPON 'Early physics run - some dectectors out'
As we first wish to add this data using the FATMEN shell, we must edit the file to include the generic-name of each dataset. To do this, we will create a file with filetype KUMAC, so that we can then execute it directly from within the FATMEN shell. First, we include a command to change the current directory to //CERN/OPAL/LEPD/RAWD/P51989. This is done using the command cd. Note that there is no need to type the //CERN/OPAL as we enter at this level by default. Then, between the file sequence number and dataset name, we insert the generic-name. For simplity, we will refer to these files as FILE1 to FILE10. We also add the command ADD/TAPE at the beginning of each line to instruct the FATMEN shell to add this information to the database.
Resultant file TESTFAT KUMAC (or testfat.kumac)
cd lepd/rawd/p51989 ADD/TAPE JS1001 CIN136 1 TAPE1 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1002 CIN137 1 TAPE2 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1003 CIN138 1 TAPE3 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1004 CIN139 1 TAPE4 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1005 CIN140 1 TAPE5 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1006 CIN141 1 TAPE6 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1007 CIN142 1 TAPE7 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1008 CIN143 1 TAPE8 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1009 CIN144 1 TAPE9 RAWDATA FX 0 VXOPON 'Early physics run - some detectors out' ADD/TAPE JS1010 CIN145 1 TAPE10 RAWDATA FX 0 VXOPON 'Early physics run -some dectectors out'
We can now procede to run this macro, by typing the following commands:
FM EXEC TESTFAT END
Now that these files have been catalogued, we can look at the catalogue information using the ls command. This command allows us to list the -- various files and there attributes. For example, the command ls tape% -a ----------- would list all details (option a) for the files tape1-tape9. Tape10 would not be listed as the % character only matches against a single character. We must use the syntax ls tape* -a to see also this file. -----------
Of perhaps more interest is the ability to be able to access the data itself by using the generic name. This is performed by using the find command. Thus, we could type the commands
fm cd lepd/rawd/p51989 find tape7 iofile13 end
This would initiate an input tape stage of the volume matching the generic name
//CERN/OPAL/LEPD/RAWD/P51989/TAPE7
Once the stage has completed, a FORTRAN program can read the data on OFILE13.
The following example shows how data may be added to the catalogue using the FORTRAN interface. The FMADDD and FMADDT routines provide the same functionality as the shell ADD/DISK and ADD/TAPE commands.
Adding information to the catalogue
PROGRAM ADDTEST
CHARACTER*256 GENAME,DSN
CHARACTER*80 COMM
CHARACTER*8 HOST,CHUSER
CHARACTER*4 FFORM,RECFM,CHOPT
*
* Sequence FATPARA from PATCH FATCDES on FATMEN pam
*
+CDE,FATPARA.
CALL FMSTRT(1,2,'//CERN/CNDIV',IRC)
CALL FMLOGL(3)
GENAME = '//CERN/CNDIV/JAMIE/ULF'
DSN = 'BOX.SET'
FFORM = 'AS'
HOST = 'CERNVM'
RECFM = 'V'
COMM = 'ADDED VIA NEW ADDTEST FORTRAN'
CHOPT = 'N'
*
* CHOPT = N : do not add this entry to the catalogue
* this allows us to modify other fields before sending
* the update to the server.
*
CALL FMADDD(GENAME,DSN,FFORM,0,HOST,RECFM,80,11,483,2,COMM,
+ IVECT,CHOPT,IRC)
*
* Update user file format - this field is not accessible
* directly via FMADDD
*
* Two ways of doing this are shown.
*
+SELF,IF=PEEK.
*
* Get contents of bank into vector IVECT
*
CALL FMPEEK(GENAME,IVECT,' ',IRC)
*
* Update user file format
*
CALL UCTOH('NFF',IVECT(MFUTFA),4,3)
*
* Copy vector back and update catalogue
*
CALL FMPOKE(GENAME,IVECT,'P',IRC)
+SELF,IF=-PEEK.
CALL FMPUTC(-1,'NFF',MFUTFA,3,IRC)
*
* Now update the catalogue
* Options: N - ignore IVECT
* P - add entry via FMPUT
*
CALL FMPOKE(GENAME,IVECT,'NP',IRC)
+SELF.
CALL FMEND(IRC)
PRINT *,'Return code ',IRC,' from FMEND'
*
END