For the Impatient
This section will give you the essential steps to setup a new SODAR Core based project. We will link to the parts of the manual where they were taken from such that you can read more in depth there.
See It In Action
We have developed the following data management and analysis web applications using SODAR Core. Although there only is a public demo available for VarFish at this time, the source code of the applications demonstrate how to use SODAR Core in complex web applications.
- VarFish
is a web-based tool for the analysis of variants. It showcases how to build a complex data warehousing and data analysis web appliction using SODAR Core. More details are described in the NAR Web Server Issue publication (doi:10.1093/nar/gkaa241). The source code can be found on github.com/bihealth/varfish-server. A demo is available at varfish-demo.bihealth.org.
- DigestiFlow
is a web-based data system for the management and demultiplexing of Illumina Flow Cells. It further implements various tools for sanity checking Illumina sample sheets and quality control (e.g., comparing barcode adapter sequence and actual sequence present in the sequencer output). You can find out more in our publication in Bioinformatics (doi:10.1093/bioinformatics/btz850). The source code can be found on github.com/bihealth/digestiflow-server. There currently is no public demo instance yet.
- Kiosc
is a web application that allows to build scheduler Docker containers for “data science” apps and dashboards. There currently is no public demo instance yet.
Prerequisites
- Linux / Mac
We use bash syntax on a Unix system and assume that you can adjust this to your system when needed.
- PostgreSQL
Please install version 9.6 or above. We assume that you have access to the
postgres
user or some other administrative user.- Development Essentials
We assume that you have
git
, Python 3.8 or above, and other essential tools installed. If you are using a mainstream Unix-like distribution (Mac qualifies) then you should be good to go.
Isolate Python Environment
If you use virtualenv, please create a new virtual environment for the project and activate it. Otherwise, follow the previous link and do this now or you can follow along us using conda.
The following creates a new Miniconda installation on 64 bit Linux or Mac. The Miniconda website has URLs to more.
# Linux
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh -b -p ~/miniconda3
$ source ~/miniconda3/bin/activate
$ conda install -y python=3.8
# Mac
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
$ bash Miniconda3-latest-MacOSX-x86_64.sh -b -p ~/miniconda3
$ source ~/miniconda3/bin/activate
$ conda install -y python=3.8
For activating the conda installation, use source ~/miniconda3/bin/activate
.
Install SODAR Core
We simply use pip
for this:
# you must have your miniconda3 install sourced, skip if done already
$ source ~/miniconda3/bin/activate
$ pip install django-sodar-core==0.10.8
Download Example Site
We maintain a Git repository with a django project using the latest SODAR Core version here on GitHub: sodar-django-site. We will use this to get hit the ground running. See Projectroles Integration on other ways to get started with SODAR Core.
$ git clone https://github.com/bihealth/sodar-django-site.git sodar-django-site
$ cd sodar-django-site
From here on, we assume that you are located (a) within the
sodar-django-site
directory and (b) have done
source ~/miniconda3/bin/activate
such that which python
shows
~/miniconda3/bin/python
.
To complete this step install the development requirements.
$ pip install -r requirements/local.txt
Configure Environment
The next step is to perform some configuration.
SODAR Core is built on the 12 factor app principles.
Configuration is done using environment variables.
For development, they are read from the .env
file in your
sodar-django-site
checkout.
We are shipping an example setting file that you should copy and then edit.
$ cp env.example .env
# now edit .env
To start out, it will be sufficient to make sure you can connect to the database. The default value for this is shown below.
DATABASE_URL="postgres://sodar-django-site:sodar-django-site@127.0.0.1/sodar-django-site"
To keep it simple, you can use the following commands to create the correct database, user, and set the password.
$ sudo -u postgres createuser -ds sodar-django-site -W
[sudo] password for USER: <enter your password>
Password: <enter: sodar-django-site>
$ sudo -u postgres createdb --owner=sodar-django-site sodar-django-site
Now, we have to make sure that the environment file is read:
$ sed -ie "s/^READ_DOT_ENV_FILE.*/READ_DOT_ENV_FILE = env.bool('DJANGO_READ_DOT_ENV_FILE', default=True)/" config/settings/base.py
Database Initialization
For the final steps, you will initialize the database…
$ python manage.py migrate
… and create a new admin user …
$ python manage.py createsuperuser
Username: root
Email address: root@example.com
Password:
Password (again):
Superuser created successfully.
The First Login
Now, start the server, and you can then visit http://127.0.0.1:8000/login/?next=/
$ make serve
python manage.py runserver --settings=config.settings.local
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
July 01, 2021 - 13:06:06
Django version 3.2.5, using settings 'config.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
You should see the following:
Now, login with your superuser and you should see the following:
Click the little user icon on the top right to access the django admin (here is where you can create more users, for example) but also the preconfigured Site Apps adminalerts, siteinfo, userprofile and configuration for remote sites. The plus button on the left allows to create new categories and projects.
Now might also be a good time to read up more on the projectroles app as this is the fundamental app for most further development.
The First Project
You cannot create projects on the root level but you have to create a new category first (collections of projects). Use the “create category” button on the left to create a “example category” first, then create an “example project” within. The project details view should look as follows.
Note that the site search already works, so typing “example” into the text field on the top and clicking “search” will return your example project. The project overview shows the “project home card” for the installed project apps filefolders, timeline, and bgjobs. Usually, the latest five entries are shown here.
Note
The filesfolders app is an example of the data management application of SODAR Core based apps. You can easily imagine a more advanced module/app that not only allows tagging of files but more structuring data and meta data more strongly.
SODAR Core was extracted as a re-useable library of component from our (not yet released) SODAR project that allows the management of structured meta data of experiments and files with data from such experiments.
Go ahead and try out the filesfolders app by clicking the “small files” icon on the left. After creating folders and uploading a few files, you will see a trace of actions in the timeline app:
Note
By default, sodar-django-site
will store the files in the PostgreSQL
database but you can easily configure it to use other storage backends,
e.g., the S3 protocol, with the
django-storage
package but this goes beyond this documentation.
Also, you will be able to find your uploaded file by name through the search box. Note that any app that you write can easily provide all the integrations with the SODAR Core framework (your apps are no different than the built-in ones). Just have a look how we did it in the apps shipping with SODAR Core.
Summary
Here is a quick summary on how SODAR Core interacts with the built-in and user apps:
At the lower most level all content is managed in projects which themselves can be assigned into categories.
Project apps can provide new content types that can be put into projects. For example, the filesfolders app allows you to store files, folders, and assign meta data to them. As another example, the timelines app stores events that occured in a project, and other apps such as the filesfolders app can register their events with it.
SODAR Core defines several plugin extension points that your apps can implement and make their content findable, for example.
Site apps allow to provide features independent of a project. For example, the userprofile app allows to access user settings and the adminalerts app allows to post global notifications.
Going on From Here
You can now start exploring your
sodar-django-site
and play around with it.You can read the :ref`user_stories` section to learn how SODAR Core based applications are built.
Continue reading Getting Started for a more comprehensive documentation and walk-through of SODAR Core and its apps.
Have a look at the web apps developed by us that are using SODAR Core as shown in the See It In Action section.