Create a table of metadata for each artifact to be stored in the STRS application repository
Usage
The getArtifactMetadata Python script is used to create a consistent table showing the metadata for each file in a directory tree. The metadata will be stored in the STRS application repository, in addition to the artifacts themselves. The metadata displayed is the privately searchable portion of the metadata.
The getArtifactMetadata Python script is used to create a CSV file for submittal because of Excel’s ability to process CSV files and edit them. If Excel is not available to the originator, the originator may use other database or similar programs for checking the data.
Setup
- Install Python. See Python download site. The script was tested against Python 2.7 and Python 3.2. Since there were residual errors when handling special characters in Python 2.7, it is recommended that the user install and use Python 3.2 or above.
- Create/find a working directory to store scripts and execution results for upload and download. This working directory must be outside the directory tree to be analyzed.
- Save the getArtifactMetadata.python.txt script into that working directory.
Create Web Page of Directory Tree Contents
- Find base of the directory tree of the application.
- In a command window, execute the python script getArtifactMetadata.python.txt with the input directory being the base of the directory tree and the output being the new file(s).
usage: python getArtifactMetadata.python.txt [-h] [-t TITLE] [-s SUBDIRECTORY] directory
positional arguments: directory Path to files to be analyzed. optional arguments: -h, --help show this help message and exit. -t TITLE, --title TITLE Title used in CSV output. The CSV title should be project and application name. If not entered here, the user is prompted during execution. -s SUBDIRECTORY, --subdirectory SUBDIRECTORY Subdirectory to output CSV file. If not specified, the current directory is used.
- Regardless of the machine type, when the script is executed, it will prompt for any missing title and whether to prompt for further file information. The CSV title should be the same as used elsewhere to identify the application. If the user agrees to prompt for further information, for each file, the user is prompted for the type of reviewer/SME for compliance testing, for purpose/functionality, for distribution dependencies (1), and for programming dependencies. If the user does not prompt for further information, a TBD value will be filled in for those entries. The user may also want TBD values entered when the user has a lot of files and wants to edit this metadata later. In which case, the user may edit the TBD data with Excel, for example.
- In a PC command window, the following was executed:
cd C:path\SoftwareDefinedRadio\STRS Repository C:\python27\python getArtifactMetadata.python.txt . -t "The title for app." directory = . subdirectory = => C:path\SoftwareDefinedRadio\STRS Repository csvTitle = The title for app. Do you want to prompt for file information interactively? (Y, N, Q, or ?): n ... ***************************************************************************** *** Created CSV file: Artifact.20130926154030.csv, containing 121 items. *** *****************************************************************************
Note that if there are any spaces in the command line options (e.g. title), a single set of quotes is sufficient to make python get the command line arguments as a single string.
- On Windows using one set of quotes around a title containing spaces, the quotes disappear but title with spaces is parsed properly.
- On Windows using mixed quotes around a title containing spaces (i.e. using both single and double quotes, for example, “‘a b c'”), any double quotes disappear but the title is parsed properly.
- In a PC command window, the following was executed:
cd "C:path\SoftwareDefinedRadio\STRS Repository" C:\python27\python getArtifactMetadata.python.txt . directory = . subdirectory = => C:path\SoftwareDefinedRadio\STRS Repository csvTitle = Enter title of CSV file (usually project and application name) or Q or ?: The title for App. csvTitle = The title for app. Do you want to prompt for file information interactively? (Y, N, Q, or ?): y Now = Sep 27, 2013, 08:22:09 = 20130927082209 ************ ftype = cpp ********** ext = cpp ********** purpose/functionality for file1.cpp (or " or "* or Q or TBD or TBD* or ?): WF application Distribution Dependencies (1) for file1.cpp (or " or "* or Q or TBD or TBD* or ?): See SRA for SUA Programming Dependencies (2) for file1.cpp (or " or "* or Q or TBD or TBD* or ?): file1.h ... ***************************************************************************** *** Created CSV file: Artifact.20130927082209.csv, containing 121 items. *** *****************************************************************************
Note that the for the items missing from the command line, the subdirectory where the output files are to be saved is the current directory, and the other items are prompted for. The output CSV file names are generated containing the word Artifact followed by the date and time.
- In a Linux command window, the following was executed:
cd $HOME/strs python27 getArtifactMetadata.python.txt STRS_Architecture/WFCCN -t "WFCCN" ... *************************************************************************** *** Created CSV file: Artifact.20130927082209.csv, containing 9 items. *** ***************************************************************************
- In a Mac terminal command window, the following was executed:
cd $HOME/strs python getArtifactMetadata.python.txt STRS_Architecture/WFCCN -t "WFCCN" ... *************************************************************************** *** Created CSV file: Artifact.20130927082209.csv, containing 9 items. *** ***************************************************************************
Note that if there are any spaces in the command line options (e.g. title), a single set of quotes are not sufficient to make python get the command line arguments as a single string.
- Escaping the spaces in the command line did not work.
- Substituting for each space in the command line title worked by letting the browser transform them into the spaces in the title where the Linux Bourne shell and python see continuous characters. That wouldn’t work for a file name.
- On Linux, using mixed quotes around a title or file name with spaces works. One set of quotes is consumed by the Linux Bourne shell and the remaining set is recognized by python, where the quotes disappear when using argparse and it can’t find the end of the title or file name with spaces. However, only the inner set of quotes disappear which allowed a work-around where I parsed the command line and put the pieces back together until the quotes matched.
- Check the CSV file for TBD. If the CSV file is edited with Excel, it must be resaved as a CSV file.
Encrypt
Send the resulting CSV file to the STRS repository manager so that it can be incorporated into the STRS Application Repository. Please identify the application by name so that the file can be associated with the correct application entry. Please use encryption, if necessary. If the contents should be encrypted, encrypt and send to the STRS repository manager. The following are some suggested methods to send encrypted metadata:
- Send the file via LFT
- Attach the file to an email,encrypt the email,and send it
- Encrypt the file and email the encrypted file as an attachment.
To send the file via NOMAD Large File Transfer (LFT), login into the following web site https://transfer.ndc.nasa.gov. Neither PKI Entrust nor LFT can encrypt to the STRS repository manager’s mailing list; therefore, if using either method, the file destination is specified as the substitute mailing link provided in the web form output or described in the STRS web site at https://strs.grc.nasa.gov/repository/. External users can only use LFT to send to internal NASA users. If you are a user external to NASA, you need send an email to the STRS repository manager requesting access to LFT web site. You will receive an invitation email. Click on the link in the invitation email to access the LFT web site and register for an account.
Neither PKI Entrust nor NOMAD Large File Transfer (LFT) can encrypt to a mailing list. External users can only use LFT to send to internal NASA users.
- If using PKI Entrust, click on this substitute mailing link instead of using the STRS repository manager’s mailing list. Do not send unencryption credentials in the same email.
- If you are an internal user using LFT, send metadata to Tina.G.Grzincic@nasa.gov and Janette.C.Briones@nasa.gov instead of using the STRS repository manager’s mailing list.
- If you are a user external to NASA, send an email to the STRS repository manager requesting access to NOMAD Large File Transfer (LFT) website. Once approved, you will receive an invitation email. Click on the link in the invitation email to access the Large File Transfer website and register for an account. Further instructions are located on the LFT website.
The NOMAD Large File Transfer service is now FIPS 140-2 certified for Data-in-Transit (DIT) and Data-at-Rest (DAR). This satisfies the NASA policy requirement to encrypt sensitive but unclassified (SBU) information while in transit – therefore you do not need to encrypt a file before sending it through LFT.