Data Management
Overview - First Draft
The Genetics and Evolution Data Management Plan provides guidance to:
- Manage data storage and archive
- Document our datasets, projects, analyses
- Define our resources
1. Data Storage Options
Considerations for organizing data storage:
- What are the data that you are storing?
- Hot, Cold, Warm?
- Current, Future access needs?
1.2 Google Cloud Buckets
- CB Genetics Bucket
- Use Cases:
- Finished projects that no longer need to be downloaded
- Archive data that no longer needs to be read or downloaded
- Data / files that will be worked with in Google Workstation
- Protocol:
- Transfer data with
gcloud storagecommand suite (gcloud storage rsync) - Instructions incoming
- Naming convention: Name your folder
ProjectName_Analysis_DateorProjectName_Archive_Datedepending on whether you have an active project being worked on in Google Workstation (Analysis) or you are storing an archived dataset or analysis (Archive)
- Transfer data with
- Limitations: Costs to read and download the data - new resource / unknown how pay structure will work
1.3 Sedna
- Use Cases: Active storage for data that is currently being used
- Protocol: Put path to your data in inventory spreadsheet (e.g.,
Sedna- /home/user/radseq/data) - Limitations: Space is limited, not a long-term location for archived data
1.4 Local Instrument Storage
- Use Cases: Storage for data directly from machine (data generation)
- Protocol: Indicate folder name and instrument in Local Instrument column
- Limitations: Space is limited, not a long-term location for archived data, everyone on the team has access and someone could delete your data
1.5 Personal Computer
- Use Cases: Active storage for data that is currently being used
- Protocol: Put “personal computer” in inventory spreadsheet
- Limitations: No one else can access, space is limited, not a long-term location for archived data
1.6 NCBI / SRA / Public Repositories
- Use Cases: Archive for raw sequencing data and relevant metadata
- Protocol: Upload to SRA, delay publishing for 5 years if desired
- All papers using genomic data will have sequences uploaded to SRA - this is our best long-term archive option
| Primary storage location | Secondary storage locations |
|---|---|
| Shared GDrive (NOT personal MyDrive) Google Bucket SRA (NCBI) |
Shared GDrive (NOT personal MyDrive) Google Bucket SRA (NCBI) Sedna QNAP |
| Option | Pros | Cons |
|---|---|---|
| Shared Drive | Accessible to team members | Limited File Number Bottleneck transfer size |
| Google Bucket | Not very accessible for analysis (unless using Google Cloud) Cost Associated with Reading and Downloading |
|
| QNAP | Already exists off of sequencer | Does not last forever |
| Personal Computer | Free Easy Access for analysis |
Not accessible to others |
| Sedna | Free | Space can be limited long term |
| SRA | Free Accessible Complies with public availability of data |
2. Data Transfer
2.1 Rclone
Rclone requires configuration the first time you use it. On your local computer, just download rclone and put it in a folder on your computer (documents, desktop, downloads, etc.). See Instructions to set up Rclone on your local machine or Sedna.
Usage:
```bash rclone copy /path/to/file.txt google:destination/folder/“quotes for folders with spaces”