Skip to main content

Data Management

Overview - First Draft

The Genetics and Evolution Data Management Plan provides guidance to:

  • Manage data storage and archive
  • Document our datasets, projects, analyses
  • Define our resources

1. Data Storage Options

Considerations for organizing data storage:

  • What are the data that you are storing?
  • Hot, Cold, Warm?
  • Current, Future access needs?

1.1 Shared Drives

  • Use Cases:
  • Protocol:
  • Limitations:
    • Limit on number of files, folders, and shortcuts
    • IF SYNCING WITH COMPUTER:

1.2 Google Cloud Buckets

  • CB Genetics Bucket
  • Use Cases:
    • Finished projects that no longer need to be downloaded
    • Archive data that no longer needs to be read or downloaded
    • Data / files that will be worked with in Google Workstation
  • Protocol:
    • Transfer data with gcloud storage command suite (gcloud storage rsync)
    • Instructions incoming
    • Naming convention: Name your folder ProjectName_Analysis_Date or ProjectName_Archive_Date depending on whether you have an active project being worked on in Google Workstation (Analysis) or you are storing an archived dataset or analysis (Archive)
  • Limitations: Costs to read and download the data - new resource / unknown how pay structure will work

1.3 Sedna

  • Use Cases: Active storage for data that is currently being used
  • Protocol: Put path to your data in inventory spreadsheet (e.g., Sedna- /home/user/radseq/data)
  • Limitations: Space is limited, not a long-term location for archived data

1.4 Local Instrument Storage

  • Use Cases: Storage for data directly from machine (data generation)
  • Protocol: Indicate folder name and instrument in Local Instrument column
  • Limitations: Space is limited, not a long-term location for archived data, everyone on the team has access and someone could delete your data

1.5 Personal Computer

  • Use Cases: Active storage for data that is currently being used
  • Protocol: Put “personal computer” in inventory spreadsheet
  • Limitations: No one else can access, space is limited, not a long-term location for archived data

1.6 NCBI / SRA / Public Repositories

  • Use Cases: Archive for raw sequencing data and relevant metadata
  • Protocol: Upload to SRA, delay publishing for 5 years if desired
  • All papers using genomic data will have sequences uploaded to SRA - this is our best long-term archive option
Primary storage location Secondary storage locations
Shared GDrive (NOT personal MyDrive)
Google Bucket
SRA (NCBI)
Shared GDrive (NOT personal MyDrive)
Google Bucket
SRA (NCBI)
Sedna
QNAP
Option Pros Cons
Shared Drive Accessible to team members Limited File Number
Bottleneck transfer size
Google Bucket Not very accessible for analysis (unless using Google Cloud)
Cost Associated with Reading and Downloading
QNAP Already exists off of sequencer Does not last forever
Personal Computer Free
Easy Access for analysis
Not accessible to others
Sedna Free Space can be limited long term
SRA Free
Accessible
Complies with public availability of data

2. Data Transfer

2.1 Rclone

Rclone requires configuration the first time you use it. On your local computer, just download rclone and put it in a folder on your computer (documents, desktop, downloads, etc.). See Instructions to set up Rclone on your local machine or Sedna.

Usage:

```bash rclone copy /path/to/file.txt google:destination/folder/“quotes for folders with spaces”