For the Impatient
This section will give you the essential steps to setup a new SODAR Core based project. We will link to the parts of the manual where they were taken from such that you can read more in depth there.
See It In Action
We have developed the following data management and analysis web applications using SODAR Core. Although there only is a public demo available for VarFish at this time, the source code of the applications demonstrate how to use SODAR Core in complex web applications.
- SODAR
The system for Omics data access and retrieval. This system is used to model study metadata and manage associated data files in different Omics research projects. SODAR Core was created by separating re-usable research project management components from the SODAR project.
- VarFish
A web-based tool for the analysis of variants. It showcases how to build a complex data warehousing and data analysis web appliction using SODAR Core. More details are described in the NAR Web Server Issue publication (doi:10.1093/nar/gkaa241). The source code can be found on github.com/bihealth/varfish-server. A demo is available at varfish-demo.bihealth.org.
- DigestiFlow
A web-based data system for the management and demultiplexing of Illumina Flow Cells. It further implements various tools for sanity checking Illumina sample sheets and quality control (e.g., comparing barcode adapter sequence and actual sequence present in the sequencer output). You can find out more in our publication in Bioinformatics (doi:10.1093/bioinformatics/btz850). The source code can be found on github.com/bihealth/digestiflow-server. There currently is no public demo instance yet.
- Kiosc
A web application that allows to build scheduler Docker containers for “data science” apps and dashboards. There currently is no public demo instance yet.
Prerequisites
- Linux / Mac
We use bash syntax on a Unix system and assume that you can adjust this to your system when needed.
- PostgreSQL
Please install version 9.6 or above. We assume that you have access to the
postgres
user or some other administrative user.- Development Essentials
We assume that you have
git
, Python 3.8 or above, and other essential tools installed. If you are using a mainstream Unix-like distribution (Mac qualifies) then you should be good to go.
Isolate Python Environment
If you use virtualenv, please create a new virtual environment for the project and activate it. Otherwise, follow the previous link and do this now or you can follow along us using conda.
The following creates a new Miniconda installation on 64 bit Linux or Mac. The Miniconda website has URLs to more.
# Linux
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh -b -p ~/miniconda3
$ source ~/miniconda3/bin/activate
$ conda install -y python=3.8
# Mac
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
$ bash Miniconda3-latest-MacOSX-x86_64.sh -b -p ~/miniconda3
$ source ~/miniconda3/bin/activate
$ conda install -y python=3.8
For activating the conda installation, use source ~/miniconda3/bin/activate
.
Download Example Site
The recommended way to get started with SODAR Core is to download and set up the example Django site, which installs SODAR Core and also sets up a default web site around it.
We maintain a Git repository with a django project using the latest SODAR Core version here on GitHub: sodar-django-site. See Projectroles Integration on other ways to get started with SODAR Core.
$ git clone https://github.com/bihealth/sodar-django-site.git sodar-django-site
$ cd sodar-django-site
From here on, we assume that you are located (a) within the
sodar-django-site
directory and (b) have done
source ~/miniconda3/bin/activate
such that which python
shows
~/miniconda3/bin/python
.
To complete this step install the development requirements.
# you must have your miniconda3 install sourced, skip if done already
$ source ~/miniconda3/bin/activate
$ pip install -r requirements/local.txt
Configure Environment
The next step is to perform some configuration.
SODAR Core is built on the 12 factor app principles.
Configuration is done using environment variables.
For development, they are read from the .env
file in your
sodar-django-site
checkout.
We are shipping an example setting file that you should copy and then edit.
$ cp env.example .env
# now edit .env
To start out, it will be sufficient to make sure you can connect to the database. The default value for this is shown below.
DATABASE_URL="postgres://sodar-django-site:sodar-django-site@127.0.0.1/sodar-django-site"
You can use the following commands to create the correct database, user, and set
the password. Alternatively, you can run the utility/setup_database.sh
script and fill out the values as prompted.
$ sudo -u postgres createuser -ds sodar-django-site -W
[sudo] password for USER: <enter your password>
Password: <enter: sodar-django-site>
$ sudo -u postgres createdb --owner=sodar-django-site sodar-django-site
Now, we have to make sure that the environment file is read:
$ sed -ie "s/^READ_DOT_ENV_FILE.*/READ_DOT_ENV_FILE = env.bool('DJANGO_READ_DOT_ENV_FILE', default=True)/" config/settings/base.py
Database Initialization
For the final steps, you will need to initialize the database:
$ python manage.py migrate
Finally, create an admin user:
$ python manage.py createsuperuser
Username: admin
Email address: admin@example.com
Password:
Password (again):
Superuser created successfully.
The First Login
To access the site, start the server with make serve
and use your web
browser to navigate to the following URL: http://127.0.0.1:8000
$ make serve
python manage.py runserver --settings=config.settings.local
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
July 01, 2021 - 13:06:06
Django version 3.2.5, using settings 'config.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
You should see the following:
Login with the superuser account you created. Afterwards you are redirected to your home view:
By clicking the user icon on the top right corner you can access the Django admin (where you can create more users, for example) but also the preconfigured site apps Adminalerts, Siteinfo, Userprofile and configuration for remote sites. The Create Category link on the left hand sidebar allows you to create new categories.
Now might also be a good time to read up more on the Projectroles app as this is the fundamental app for most further development.
The First Project
Creating projects on the root level is not allowed by default, so you have to create a new category first. A category is a collection of projects and/or subcategories. First click the Create Category link on the sidebar to create an “Example Category”. This takes you to the view of the newly created category. Next, click the Create a Project or Category link to create an “Example Project” within. The project details view should look as follows.
At this point you can test the search functionality. Typing “example” into the text field on the top bar and clicking Search will return your example project. The project overview shows the overview card for installed project apps Filefolders, Timeline, and Bgjobs. Usually, the five most recent entries are shown here.
Note
The Filesfolders app is an example of the data management application of SODAR Core based apps. You can easily imagine a more advanced module/app that not only allows tagging of files but more structuring data and meta data more strongly.
Go ahead and try out the Filesfolders app by clicking the Small Files link on the sidebar. After creating folders and uploading a few files, you will see a trace of actions in the Timeline app:
Note
By default, sodar-django-site
will store the files in the PostgreSQL
database. You can easily configure it to use other storage backends,
e.g., the S3 protocol, with the
django-storage
package, but is beyond the scope of this documentation.
You will now also be able to find your uploaded file by name through the search box. Note that any app that you write can easily provide all the integrations with the SODAR Core framework, as your apps are no different than the built-in ones. Just have a look how we did it in the apps shipping with SODAR Core.
Summary
Here is a quick summary on how SODAR Core interacts with the built-in and user apps:
At the lower most level all content is managed in projects which themselves can be assigned into categories.
Project apps can provide new content types that can be put into projects. For example, the Filesfolders app allows you to store files, folders, and assign meta data to them. As another example, the timelines app stores events that occured in a project, and other apps such as the Filesfolders app can register their events with it.
SODAR Core defines several plugin extension points that your apps can implement and make their content findable, for example.
Site apps allow to provide features independent of a project. For example, the userprofile app allows to access user settings and the adminalerts app allows to post global notifications.
Going on From Here
You can now start exploring your
sodar-django-site
and play around with it.You can read the User Stories section to learn how SODAR Core based applications are built.
Continue reading Getting Started for a more comprehensive documentation and walk-through of SODAR Core and its apps.
Have a look at the web apps developed by us that are using SODAR Core as shown in the See It In Action section.