Architecture

This document uses the C4 Model for visualising the architecture of grand-challenge.org. This covers System Context, Containers (Applications/Data Stores), and Components (Modules/Libraries).

Ubiquitous Language

Users

Many different users and roles are supported on grand-challenge.org.

Researcher
A Researcher is a user who wants to manage medical imaging data and reports, collect annotations on medical imaging data, create a challenge for the data science community to generate solutions to clinical problems, and objectively evaluate the submitted algorithms to challenges.
Data Scientist
A Data Scientist is a user who participates in a challenge, downloads the training medical imaging data & annotations for a challenge, uploads algorithms for a challenge, and makes those algorithms available for clinicians to execute.
Clinician
A Clinician is a user who uses a workstation to learn how to annotate or read medical imaging data, uses a workstation to make annotations to medical imaging data, and uploads new medical imaging data for execution by algorithms.

System Context

An overview of the grand-challenge.org system, its users, and its system dependencies.

@startuml
!includeurl https://raw.githubusercontent.com/RicardoNiepel/C4-PlantUML/release/1-0/C4_Context.puml

LAYOUT_TOP_DOWN
LAYOUT_WITH_LEGEND()

title System Context diagram for grand-challenge.org

Person(clinicians, "Clinician", "User who annotates data, or has new clinical data for algorithm validation.")
Person(challenge_orgs, "Researcher", "User who poses challenges and compares state of the art algorithms.")
Person(data_scientists, "Data Scientist", "User who develops machine learning algorithms.")

System(grand_challenge, "grand-challenge.org", "Allows users to gather data, run workstations, train clinicians, annotate data, pose a challenge, evaluate solutions and deploy algorithms.")

System_Ext(grand_challenge_bridge, "Grand Challenge Bridge", "Connects to hospital systems, de-itentifies data, uploads data for algorithm execution and returns algorithm results.")
System_Ext(oauth, "Authentication", "Authenticates users through external providers (Google, GitHub).")
System_Ext(monitoring, "Monitoring and Alerting", "Monitors the system and sends alerts (Sentry, Prometheus).")
System_Ext(cdn, "Content Delivery Network", "Stores and serves static data (S3, CloudFront).")
System_Ext(mail, "E-mail", "Sends emails to users (SES).")

Rel(challenge_orgs, grand_challenge, "Uses")
Rel(data_scientists, grand_challenge, "Uses")

Rel(clinicians, grand_challenge, "Uses")
Rel(clinicians, grand_challenge_bridge, "Sends data from PACS to")
Rel(grand_challenge_bridge, grand_challenge, "Uploads data to")

Rel(grand_challenge, oauth, "Authenticates users against")
Rel(grand_challenge, mail, "Sends emails through")

Rel_Neighbor(grand_challenge, cdn, "Sends static data to")
Rel_Neighbor(monitoring, grand_challenge, "Scrapes monitoring endpoints of")
@enduml

Containers

The overall shape of the architecture and technology choices. Note: a container is a separately deployable application or data store, not necessarily a Docker container.

@startuml
!includeurl https://raw.githubusercontent.com/RicardoNiepel/C4-PlantUML/release/1-0/C4_Container.puml

LAYOUT_TOP_DOWN
LAYOUT_WITH_LEGEND()

title Container diagram for grand-challenge.org

Person(clinicians, "Clinician", "User who annotates data, or has new clinical data for algorithm validation.")
Person(challenge_orgs, "Researcher", "User who poses challenges and compares state of the art algorithms.")
Person(data_scientists, "Data Scientist", "User who develops machine learning algorithms.")

System_Boundary(s1, "grand-challenge.org") {
    Container(gc_web_app, "Web Application", "Nginx, Django", "Provides the grand challenge functionality via HTML and a REST API.")
    Container(gc_algorithm_worker, "Algorithm and Evaluation Worker", "Django, Docker, Celery, Redis", "Executes algorithm and evaluation container images on medical imaging data.")
    Container(gc_workstation_worker, "Workstation Worker", "Nginx, Django, Docker, Traefik, Celery, Redis", "Instantiates and proxies workstation container images for users.")

    Container(workstation_instance, "Web Workstation", "SPA, MeVisLab, VueJS, Docker", "Provides a web based workstation for server side rendering of medical imaging data. 1 container image per active user.")

    ContainerDb(postgres, "Database", "PostgreSQL", "Stores user information, medical imaging metadata, job results etc.")

    Container(protected_s3, "Protected Data Store", "Minio", "Stores protected medical imaging objects.")
    Container(private_s3, "Private Data Store", "Minio", "Stores private objects, container images etc.")
}

System_Ext(oauth, "Authentication", "Authenticates users through external providers (Google, GitHub).")
System_Ext(monitoring, "Monitoring and Alerting", "Monitors the system and sends alerts (Sentry, Prometheus).")
System_Ext(cdn, "Content Delivery Network", "Stores and serves static data (S3, CloudFront).")
System_Ext(mail, "E-mail", "Sends emails to users (SES).")

Rel(challenge_orgs, gc_web_app, "Uses", "HTTPS, HTML, JSON")
Rel(data_scientists, gc_web_app, "Uses", "HTTPS, HTML, JSON")
Rel(clinicians, gc_web_app, "Uses", "HTTPS, HTML")

Rel(gc_web_app, oauth, "Authenticates users against", "HTTPS, OAuth2")
Rel(s1, mail, "Sends emails through", "SMTP")
Rel(gc_web_app, cdn, "Sends static data to", "HTTPS")
Rel(monitoring, s1, "Scrapes monitoring endpoints of", "HTTPS")

Rel(gc_web_app, postgres, "Reads and writes to", "SQL")
Rel(gc_web_app, protected_s3, "Stores objects in, and proxies access to", "HTTPS, S3")
Rel(gc_web_app, private_s3, "Stores objects in", "HTTPS, S3")

Rel(gc_workstation_worker, postgres, "Reads and writes to", "SQL")
Rel(gc_workstation_worker, private_s3, "Fetches container images from", "HTTPS, S3")
Rel(gc_workstation_worker, protected_s3, "Reads objects from", "HTTPS, S3")
Rel_Neighbor(gc_workstation_worker, workstation_instance, "Instantiates and proxies", "HTTPS, Docker")
Rel_U(gc_workstation_worker, clinicians, "Provides SPA workstation to", "HTTPS, WSS, HTML, JSON")

Rel_U(gc_algorithm_worker, postgres, "Reads and writes to", "SQL")
Rel_U(gc_algorithm_worker, private_s3, "Fetches container images from", "HTTPS, S3")
Rel_U(gc_algorithm_worker, protected_s3, "Reads objects from", "HTTPS, S3")
@enduml

Components

Logical modules and their interactions with containers. Note: a component is a grouping of related functionality behind a well defined interface, not components in the sense of pipelines.