diff --git a/docs/docs/cogstack-ce/_index.md b/docs/docs/cogstack-ce/_index.md index 999bb670..122c8d1a 100644 --- a/docs/docs/cogstack-ce/_index.md +++ b/docs/docs/cogstack-ce/_index.md @@ -6,6 +6,11 @@ CogStack CE aims to show what is possible with the open source CogStack products It is installed with a one line command, and comes preconfigured with default data sets, configurations, and example dashboards already setup for you. It is built to run in Kubernetes, from local instances using Minikube, and scale up to production clusters and managed cloud environments. +## Where to start + +1. [Quickstart](./tutorial/quickstart-installation.md) for the installation +2. [Product Tour](./product-tour/) to see examples of the features + ## What is included in CogStack CE? It combines model serving, de-identification, model training, notebook-based analysis, and search tooling into one Helm release. @@ -18,10 +23,8 @@ It combines model serving, de-identification, model training, notebook-based ana | JupyterHub | Run notebooks for experimentation and end-to-end workflows. | | OpenSearch + Dashboards | Index, search, and explore operational or NLP data. | -## Where to start +![CogStack CE Architecture](./assets/cogstack-ce-architecture.png) -1. [Quickstart](./tutorial/quickstart-installation.md) for the installation -2. [Product Tour](./tutorial/end-to-end-jupyterhub.md) ## Installation and customization (reference) @@ -38,7 +41,13 @@ There are public models available, that will require a NIH profile or UMLS licen !!! tip For access to high performing models trained on real world clinical datasets, contact us + +## Architecture + +See [CogStack CE architecture](./reference.md) for a diagram of the deployed services and how the tutorials connect them. + ## Next Steps After setting up and trying CogStack Community edition, you can look into the details and wider tools in the platform -- [Getting Started with CogStack Platform](../overview/getting-started.md) \ No newline at end of file +- [Getting Started with CogStack Platform](../overview/getting-started.md) +- View the cloud deployment examples for running it in [AWS](../platform/deployment/examples/aws-kubernetes-eks.md) or [Azure](../platform/deployment/examples/azure-kubernetes-aks.md) \ No newline at end of file diff --git a/docs/docs/cogstack-ce/assets/cogstack-ce-architecture.png b/docs/docs/cogstack-ce/assets/cogstack-ce-architecture.png new file mode 100644 index 00000000..0963dcf0 Binary files /dev/null and b/docs/docs/cogstack-ce/assets/cogstack-ce-architecture.png differ diff --git a/docs/docs/cogstack-ce/product-tour.md b/docs/docs/cogstack-ce/product-tour.md index 8a6cee92..c7b54d97 100644 --- a/docs/docs/cogstack-ce/product-tour.md +++ b/docs/docs/cogstack-ce/product-tour.md @@ -24,7 +24,7 @@ In this tour, we will: ## Product Tour -### Medcat Service +### Annotate Entities - Medcat Service Open the MedCAT UI at `http://127.0.0.1:5000`, load the sample text, and click Annotate. The concepts and annotations will appear in the results panel on the right. @@ -39,7 +39,7 @@ To learn more about MedCAT and related services, see [CogStack NLP on GitHub](ht By default the cogstack community edition uses a tiny model that only learned one concept, so it wont give very good results if you try with other inputs. Check the medcat documentation for access to public models, or contact us if you want access to the best performing private models. -### AnonCAT Service +### Deidentify Notes - AnonCAT Service Open the AnonCAT UI at `http://127.0.0.1:5001`, load the sample, and click Annotate. The processed/anonymised output will be shown on the right. @@ -51,7 +51,7 @@ To learn more about AnonCAT and related services, see [CogStack NLP on GitHub](h -### MedCAT Trainer +### Train new Models - MedCAT Trainer Open MedCAT Trainer at `http://127.0.0.1:8080` and log in with your credentials. You should see it provisioned with two projects corresponding to the two services; open either project to view the documents and start annotating. @@ -61,7 +61,8 @@ For a deeper guide to projects and annotation workflows, see the [MedCATtrainer -### JupyterHub +### Integrate everything together - JupyterHub +Using JupyterHub we can now integrate all the services and get value out of your data using the CogStack Community Edition. Open JupyterHub at `http://127.0.0.1:8000` and log in with your credentials, then wait for your server to start. Open the demo notebook and run it to see the example workflow. @@ -77,7 +78,7 @@ For more details on the JupyterHub setup used here, see [CogStack Jupyter Hub on -### OpenSearch Dashboards +### Analyse your data - OpenSearch Dashboards Open OpenSearch Dashboards at `http://127.0.0.1:5601` and log in with your credentials. Browse to Dashboards and open the sample dashboard to explore the visualisations. diff --git a/docs/docs/cogstack-ce/reference.md b/docs/docs/cogstack-ce/reference.md new file mode 100644 index 00000000..d6fe15eb --- /dev/null +++ b/docs/docs/cogstack-ce/reference.md @@ -0,0 +1,105 @@ +# CogStack CE Architecture + +CogStack Community Edition deploys the open source CogStack apps into one Kubernetes release. JupyterHub is the main place you work: each user gets a JupyterLab instance with example notebooks that call the other services over HTTP. + +![CogStack CE Architecture](./assets/cogstack-ce-architecture.png) + +To best explain the architecture, we will first walk through each tutorial use case, and you can see how each service delivers part of the platform. After that, we will add how Trainer and OpenSearch are provisioned as part of the install process. + +## How the tutorials connect the services + +The tutorials included in CogStack CE all communicate over http to the deployed services. The chart is setup to automatically add environment variables to jupyter to make it easy to use (`OCR_SERVICE_URL`, `MEDCAT_SERVICE_URL`, `ANONCAT_SERVICE_URL`), by creating configmap in kubernetes. + +### OCR tutorial + +[OCR Service Tutorial](../data-engineering/ocr-service-tutorial.ipynb): Jupyter talks to OCR Service. + +The notebook sends scanned images, PDFs, and Word documents to the OCR service `/api/process` endpoint and gets extracted text back. Use this when you need to turn documents or images into text before NLP or search. + +### MedCAT service tutorial + +[MedCAT Service Tutorial](../platform/cogstack-ai/medcat-service-tutorial.ipynb): Jupyter talks to MedCAT Service and AnonCAT Service. + +The notebook sends sample clinical text to: + +- MedCAT Service `/api/process` for named entity extraction +- AnonCAT Service `/api/process` for de-identification + +Both services are MedCAT deployments: MedCAT annotates concepts, AnonCAT runs in de-identification mode. + +### End-to-end tutorial + +[MedCAT + OpenSearch end-to-end tutorial](./tutorial/medcat-opensearch-e2e.ipynb): Jupyter talks to MedCAT, then Jupyter talks to OpenSearch. + +The notebook: + +1. Indexes a sample clinical note in OpenSearch +2. Calls MedCAT Service to extract concepts from that note +3. Indexes the annotations back into OpenSearch +4. Searches the note by free text and by concept + +That is the path from unstructured text, through NLP, into searchable data. + +For how to open JupyterHub and run these notebooks, see [Using JupyterHub](./tutorial/end-to-end-jupyterhub.md). + +## MedCAT Trainer + +MedCAT Trainer is included so you can inspect annotations and do supervised learning against the same models the notebooks call. + +On startup, CogStack CE auto-provisions Trainer with two example projects. Each project uses a **remote model service** instead of uploading a model pack: + +- **Example Project** points at MedCAT Service +- **Example Deid project** points at AnonCAT Service + +Trainer sends documents to those services for annotation, so the UI stays aligned with the live CE model endpoints. + +See the [MedCAT Trainer provisioning guide](https://docs.cogstack.org/en/latest/nlp/medcat-trainer/provisioning/) for the YAML format and `useModelService` / `modelServiceUrl` options. + +Trainer also deploys its supporting stack shown in the diagram: NGINX (frontend), Django (backend), PostgreSQL, Solr, and ZooKeeper. + +## OpenSearch Dashboards + +OpenSearch stores indexed documents. OpenSearch Dashboards is the UI for exploring and visualising that data. + +On install, the CogStack CE provisioner: + +- Creates an OpenSearch index template +- Loads example documents +- Imports premade dashboards + +The dashboards and data used here were first defined in [cogstack-dashboards-demo](https://github.com/CogStack/cogstack-platform/tree/main/cogstack-dashboards-demo). + + +## Architecture Details + +### Multiple replicas + +Most CogStack services are Kubernetes Deployments, so they can run multiple instances and scale horizontally. They are designed to run in the cluster and be split across multiple nodes / VMs. + +OCR Service, MedCAT Service, and AnonCAT Service all are able to scale horizontally, even automatically using a Horizontal Pod Autoscaler. By default we just run 1 instance in the CE. + +OpenSearch is a statefulset, so it is able to run multiple instances and should be managed accordingly. By default we run 3 instances in the CogStack CE. + +### Helm subcharts + +Everything in CogStack CE is deployed as Helm subcharts of the [CogStack CE Helm chart](../platform/deployment/helm/charts/cogstack-ce-helm.md). See the documentation for each chart: + +- [MedCAT Service Helm](../platform/deployment/helm/charts/medcat-service-helm.md) (used for both MedCAT and AnonCAT) +- [OCR Service Helm](../platform/deployment/helm/charts/ocr-service-helm.md) +- [MedCAT Trainer Helm](../platform/deployment/helm/charts/medcat-trainer-helm.md) +- [CogStack JupyterHub Helm](../platform/deployment/helm/charts/cogstack-jupyterhub-helm.md) + +JupyterHub is based on the [official JupyterHub Helm chart](https://hub.jupyter.org/helm-chart/). OpenSearch and OpenSearch Dashboards also come from the [official OpenSearch Helm charts](https://opensearch-project.github.io/helm-charts/). The CogStack wrappers add defaults, images, and example notebooks on top of those upstream charts. + +### Provisioning + +Example OpenSearch data and dashboards are created by a Helm post-install hook. That hook runs custom Python defined only in Community Edition to generate synthetic documents, then imports the premade dashboards. + +This CE-only provisioner is the `CogStack CE Provisioner` box in the architecture diagram, it runs a standard python image, with custom code mounted as a configmap. The medcat trainer provisioning is handled by trainer itself, as this is a feature of the trainer helm chart and application. + + +## Next Steps + +- View the [CogStack CE Helm chart configuration](../platform/deployment/helm/charts/cogstack-ce-helm.md) for install options and customisation. +- View the cloud deployment examples for running it in [AWS](../platform/deployment/examples/aws-kubernetes-eks.md) or [Azure](../platform/deployment/examples/azure-kubernetes-aks.md) + diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index d45cdff5..e525adf2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -114,7 +114,8 @@ nav: - Tutorials: - Using JupyterHub: cogstack-ce/tutorial/end-to-end-jupyterhub.md - End To End Tutorial: cogstack-ce/tutorial/medcat-opensearch-e2e.ipynb - + - Architecture: cogstack-ce/reference.md + plugins: - multirepo: cleanup: true