Skip to content

deployment-guide.md

Latest commit

 

History

History
409 lines (293 loc) · 14.2 KB

deployment-guide.md

File metadata and controls

409 lines (293 loc) · 14.2 KB
Raw

GDI Starter Kit deployment guide

This guide provides a set of instructions on how to deploy the GDI Starter Kit in a node. It also includes the necessary steps to upload data and connect to the central services.

Index

FAIR Data Point
Beacon V2
User Portal
REMS
LS AAI
Beacon Network

Local services deployment

Prerequisites

FAIR Data Point

Installation

First, create a docker compose file (e.g. compose.yml) as this in an empty directory:

 # compose.yml

name: fdppv1

services:
  graphdb:
    image: "ontotext/graphdb:${GRAPHDB_VERSION:-10.8.8}"
    restart: no
    ports:
      - "127.0.0.1:7200:7200"
    volumes:
      - graphdbdata:/opt/graphdb/home
    healthcheck:
      test: curl http://localhost:7200/rest/monitor/infrastructure --silent --fail || exit 1
      start_interval: 3s
      start_period: 30s

  graphdb-init:
    image: "curlimages/curl:${CURL_VERSION:-latest}"
    restart: no
    environment:
      REPO_FILE: /repo.json
      REPO_NAMES: fdp
    volumes:
      - ./repo.json:/repo.json:ro
    # the following curl request returns status 201 if the repo does not exist, or status 400 otherwise
    command:
      - sh
      - -c
      - |
        for name in $$REPO_NAMES
        do
        # substitute "repo-name" by variable name in graphdb repo template
        data=$$(sed s/repo-name/$$name/g /repo.json)
        # post repo definition to graphdb rest api
        curl http://graphdb:7200/rest/repositories --verbose --silent --header "Content-Type: application/json" --data "$$data"
        done
    depends_on:
      graphdb:
        condition: service_healthy

  mongo:
    image: "mongo:${MONGO_VERSION:-8.0}"
    restart: no
    volumes:
      - mongodbdata:/data/db
    healthcheck:
      test: |
        [ $(mongosh --quiet --host localhost:27017 --eval "db.runCommand('ping').ok") = 1 ] || exit 1
      start_interval: 3s
      start_period: 30s

  fdp:
    image: "fairdata/fairdatapoint:${FDP_VERSION:-1.18}"
    restart: no
    environment:
      SERVER_PORT: 8080
      INSTANCE_CLIENTURL: http://localhost #CHANGE_ME BEFORE DOCKER COMPOSE UP TO PREFERED CUSTOM DOMAIN 
      INSTANCE_PERSISTENTURL: http://localhost #CHANGE_ME BEFORE DOCKER COMPOSE UP TO PREFERED CUSTOM DOMAIN 
      REPOSITORY_TYPE: 4
      REPOSITORY_GRAPHDB_URL: http://graphdb:7200
      REPOSITORY_GRAPHDB_REPOSITORY: fdp
    depends_on:
      mongo:
        condition: service_healthy
      graphdb-init:
        condition: service_completed_successfully
    healthcheck:
      test: wget --quiet --spider http://127.0.0.1:8080 || exit 1
      start_interval: 3s
      start_period: 30s

  fdp-client:
    image: "fairdata/fairdatapoint-client:${FDP_CLIENT_VERSION:-1.18}"
    restart: no
    ports:
      - "127.0.0.1:80:80"
    environment:
      FDP_HOST: fdp:8080
    depends_on:
      fdp:
        condition: service_healthy
    healthcheck:
      test: wget --quiet --spider http://127.0.0.1 || exit 1
      start_interval: 3s
      start_period: 30s

volumes:
  graphdbdata:
  mongodbdata:

And another file named repo.json containing the configuration for the GraphDB repository:

{
  "id": "fdp",
  "title": "",
  "type": "graphdb",
  "params": {
    "defaultNS": {
      "name": "defaultNS",
      "label": "Default namespaces for imports(';' delimited)",
      "value": ""
    },
    "imports": {
      "name": "imports",
      "label": "Imported RDF files(';' delimited)",
      "value": ""
    }
  }
}

You can now run it using:

docker compose up -d

After some time, you should be able to access FDP at http://localhost or your prefered custom domain.

There are two default user accounts, that you should change once your FDP becomes publicly available:

Here you can find a way to set up a public instance of FDP using a reverse proxy: https://docs.fairdatapoint.org/en/latest/deployment/production-deployment.html.

Kubernetes

There is a repository for shared development of Kubernetes deployment resources for FDP: https://github.com/GenomicDataInfrastructure/FAIR-Data-Point-K8s.

Connection to User Portal

Once you have deployed FDP, please communicate the host URL of your instance with LNDS team.

Metadata onboarding

Metadata validation

To onboard your data according to the latest GDI harmonised metadata model, you need to update the SHACLs. Go to GDI metadata to update them.

Large datasets

To onboard large datasets more efficiently, you can use a Jupyter notebook to automate this process. Clone the Sempyro repository and run the notebook using:

hatch run docs:jupyter lab

Then, execute for example the GDI-specific notebook to complete the upload.

Contact

For questions or support, please contact:

References

Beacon V2

Two different Beacon instances should be deployed, one for aggregated data and one for subject-level data. Both share most of the steps described below. When that is not the case, it is clarified in the text.

Requirements

System and services:

  • OS: UNIX (Linux, Mac…)
  • Architecture: amd64/arm64

Hardware:

  • CPU: >= 4
  • RAM: >= 4 GB (generally, 20% of total data size as RAM, e.g. 40 GB -> 8 GB RAM)
  • Disk space: >= 32 GB

Software:

  • Docker engine: version > 20.10.18
  • Docker-compose: version > v2.20.2

Make sure the next list of ports are free of use in your system:

  • 27017 (MongoDB)
  • 5050 (Beacon)

Deployment

First, clone the Beacon repository. You may do it in two ways:

  • Clone the original repository and checkout to the stable branch:
git clone https://github.com/EGA-archive/beacon2-pi-api.git
cd beacon2-pi-api
git checkout main
  • Clone the starter-kit repository and load the beacon2-pi-api submodule:
git clone https://github.com/GenomicDataInfrastructure/starter-kit.git
cd starter-kit
git submodule update --init beacon-v2/
cd beacon-v2

Edit the config file ri-tools/conf/conf.py by changing these variables:

  • datasetId: this variable has to match the “id” of the dataset you will relate the variants to.
  • case_level_data: change it to False for aggregated data and True for subject level data.
  • reference_genome: select your genome of reference between NCBI36, GRCh37 and GRCh38.

Only for the aggregated Beacon instance, edit the file ri-tools/pipelines/default/templates/populations.json and change the variable numberOfPopulations to the exact number of ancestries/populations you have allele frequencies for in your VCF and map how are the allele frequencies, zygosities and name of the population annotated in your VCF for each population.

Open the config file beacon/conf/conf.py and edit the variables that match your Beacon, especially beacon_id and uri variables, that are the ones that will be read by the Beacon Nework to identify and map your Beacon entry types.

Start the needed containers from the deploy folder:

docker compose up -d --build beaconprod db beacon-ri-tools

If the containers are built correctly the Beacon API will be available at http://localhost:5050/api.

Data injection

Copy your VCF files in .gz format inside the folder ri-tools/files/vcf/files_to_read/. Then, inject the variant data from the VCFs executing the next command (this step may take a few hours to finish, depending on your system resources):

docker exec ri-tools python genomicVariations_vcf.py

Inject the phenotypic data replacing individuals.json (for subject level Beacon) and datasets.json with the correct path to your file:

gzip datasets.json
gzip individuals.json # only for subject level Beacon
gunzip --stdout datasets.json.gz | docker exec -i mongoprod sh -c 'mongoimport --jsonArray --uri "mongodb://root:example@127.0.0.1:27017/beacon?authSource=admin" --collection datasets'
gunzip --stdout individuals.json.gz | docker exec -i mongoprod sh -c 'mongoimport --jsonArray --uri "mongodb://root:example@127.0.0.1:27017/beacon?authSource=admin" --collection individuals' # only for subject level Beacon

Edit the file beacon/permissions/datasets/datasets_permissions.yml adding the dataset id and its permissions. For example:

dataset_id:
  public:
    default_entry_types_granularity: record

For the API to respond fast to the queries, you have to index your database each time you inject new data::

docker exec beaconprod python -m beacon.connections.mongo.reindex

If your data collections (e.g., runs, biosamples, etc.) already contain structured metadata using ontology terms (like NCIT, UBERON, EFO...), you can extract filtering terms automatically. This will populate the /filteringTerms endpoint of your Beacon, enabling more advanced queries:

docker exec beaconprod python -m beacon.connections.mongo.extract_filtering_terms
Kubernetes

There is a repository for shared development of Kubernetes deployment resources for Beacon Allele Frequency instace: https://github.com/GenomicDataInfrastructure/BEACON-AF-K8s.

Customization

Make your Beacon your own by following these next steps:

  • Edit your instance’s metadata. Update the /info endpoint with your organization's name, description, version, and contact details.
  • Manage dataset permissions. Control which datasets are public or require authentication.
  • Enable advanced filtering.
  • Find more setting options in Configuration.

In this section of the guidelines you will find more details and examples.

Connection to the GDI Beacon Network

Follow this steps:

  • Validate the schema of your Beacon instance with Beacon Verifier v2.
  • Add your instance to the test Beacon Network by contacting Oriol Lopez-Doriga to validate its configuration.
  • Upon successful testing, the Beacon instance will be added to the GDI Beacon Network by the CRG Beacon team.

References

Contact

Beacon team at the Centre for Genomic Regulation (CRG), Barcelona, Spain:

S&I

Compute

htsget

Central services

User Portal

There is available documentation on what the service does on how it is deployed.

A staging environment instance is deployed at https://portal.staging.gdi.lu/.

A production environment instance is deployed at https://portal.gdi.lu/.

Contact

LNDS team:

REMS

REMS is integrated into the User Portal. Documentation is available at https://genomicdatainfrastructure.github.io/gdi-userportal-docs/introduction/third-party-components/rems/.

A test environment instance is deployed at https://daam.portal.testing.gdi.lu/.

A staging environment instance is deployed at https://daam.portal.staging.gdi.lu/.

Dataset Catalogue (CKAN)

CKAN is integrated into the User Portal. Documentation is available at https://genomicdatainfrastructure.github.io/gdi-userportal-docs/developer-guide/ckan/.

A test environment instance is deployed at https://catalogue.portal.testing.gdi.lu/.

A staging environment instance is deployed at https://catalogue.portal.staging.gdi.lu/.

LS AAI

For connecting your services to LS AAI read the Documentation for connecting your service to LS AAI, mainly the Instructions for relying parties.

The main user interface to LS AAI for users is LS AAI User Profile.

The main user interface for Service Provider administrators is Service Provider Registry.

The main user interface for group administrators is LS Identity and Access Management.

Contact

LS AAI team:

Beacon Network

Aggregated Beacon network

A test environment API instance is deployed at https://af-gdi-beacon-network-backend-test.ega-archive.org/beacon-network/v2.0.0.

A staging environment API instance is deployed at https://af-gdi-bn-api-demo.ega-archive.org/beacon-network/v2.0.0.

Subject-level Beacon Network

A test environment API instance is deployed at https://gdi-beacon-network-backend-test.ega-archive.org/beacon-network/v2.0.0.

A staging environment API instance is deployed at https://gdi-beacon-network-backend-demo.ega-archive.org/beacon-network/v2.0.0.

Connection with local Beacon

Follow the steps at the local Beacon section.

Contact

Beacon team