Install coge: Difference between revisions

From CoGepedia
Jump to navigation Jump to search
← Older edit
Franka1 (talk | contribs)
119 edits
No edit summary
No edit summary
 
(264 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Server with existing CoGe Installation  ==
== Installing CoGe on Ubuntu ==
 
<span style="color:green">Note: these instructions were last updated and verified on June 3rd, 2016 by mdb.</span>


=== Initial Dependencies  ===
=== Initial Dependencies  ===


Run the following command:  
Run the following command:  
<pre>sudo apt-get -y install {program}
<pre>sudo apt-get -y install {package}
</pre>  
</pre>  
<br> where {program} is each of the following:  
where {package} is each of the following:  
<pre>git
<pre>
apache2
aragorn
blast2
build-essential
checkinstall
expat
gcc-multilib
git
graphviz
imagemagick
libdb-dev
libgd2-xpm-dev
libperl-dev
libgd-gd2-perl
libconfig-yaml-perl
libssl-dev
libzmq3-dev
mysql-server
mysql-server
samtools
ncbi-blast+
ubuntu-dev-tools
ncbi-blast+-legacy
build-essential checkinstall
gcc-multilib
expat libexpat1-dev
libgd2-xpm-dev build-essential
njplot
njplot
imagemagick
graphviz
apache2
lamp-server^
phpmyadmin
phpmyadmin
python-dev
python-numpy
python-software-properties
samtools
swig
swig
ttf-mscorefonts-installer</pre>
sqlite3
Then, use:
ttf-mscorefonts-installer
<pre>sudo cpan App::cpanminus
ubuntu-dev-tools
sudo cpanm {module}
libapache-asp-perl
</pre>
libapache2-mod-perl2
where {module} is each of the following:
libapache2-mod-wsgi
<pre>AuthCas
python-pip
DBIx::Class
r-cran-plyr
DBIx::Class::Schema
r-cran-reshape2
DBD::SQLite
r-cran-ggplot2
Data::UUID
nodejs
Data::GUID
npm
XML::SAX::Expat
libboost-all-dev  (for TopHat)
XML::Parser
python-glpk
XML::Simple
glpk-utils
IPC::System::Simple
libgmp3-dev
CGI::Application::Dispatch
zimpl
CGI::Log
GD::Graph::bars
Moose
ZMQ::LibZMQ3
File::Slurp
DateTime
URI::Split
JSON
JSON::XS
</pre>
</pre>


=== Create new mysql database  ===
=== Create MySQL database  ===
 
Dump CoGe Database schema


*File download: http://data.iplantcollaborative.org/quickshare/71b18508287f9fb0/cogetable.sql (AUTO_INCREMENT removed)
Dump CoGe database schema (if using existing CoGe installation, otherwise see schema file below).


  mysqldump -d -h localhost -u root -pXXXXXXX coge | sed 's/AUTO_INCREMENT=[0-9]*\b//' &gt; cogetable.sql
  mysqldump -d -h localhost -u root -pXXXXXXX coge | sed 's/AUTO_INCREMENT=[0-9]*\b//' &gt; coge_mysql_schema.sql


Note: be sure to disable apparmour for MySQL.  
*CoGe MySQL database schema file (updated June 3rd, 2016): http://genomevolution.org/coge/coge_mysql_schema.sql
*Note: be sure to disable AppArmor for MySQL.  


Create new CoGe Database  
Create new CoGe Database  
Line 66: Line 71:
Initialize new coge database  
Initialize new coge database  


  mysql -u root -pXXXXXXXX coge &lt; cogetable.sql
  mysql -u root -pXXXXXXXX coge &lt; coge_mysql_schema.sql


Populate a few entries in the feature_type table  
Populate a few entries in the feature_type table  


*This is important because part of CoGe's code-base is keyed to feature_type_ids. This is done in order to improve performance of the system by using a feature_type_id to retrieve features of a particular type. An example is OrganismView which needs to find features of type "chromosome" in order to determine the size of a genome. The table loaded here contains [[CoGe's 10 base feature types|10 feature types]]  
*Use the table here which contains [[CoGe's 10 base feature types|10 feature types]]:   http://genomevolution.org/coge/coge_feature_types.sql
*Download file: http://data.iplantcollaborative.org/quickshare/ac8758f83c9b29b1/feature_type.sql
 
mysql -u root -pXXXXXXXXX coge &lt; feature_type.sql


Create new user for new CoGe database
mysql -u root -pXXXXXXXXX coge &lt; coge_feature_types.sql


*Want a web-user with limited write privileges and a power user to load new data
Create new MySQL user for the CoGe database


[[Image:Screen shot 2012-07-09 at 11.24.25 AM.png|thumb|right|400px]]  
[[Image:Screen shot 2012-07-09 at 11.24.25 AM.png|thumb|right|400px]]  


use mysql;
  create user 'coge'@'localhost' IDENTIFIED BY 'XXXXXX';
  create user 'coge'@'localhost' IDENTIFIED BY 'XXXXXX';
  grant all privileges on coge.* to coge;
  grant all privileges on coge.* to coge;
create user 'coge_web'@'localhost' IDENTIFIED BY 'XXXXXX';
grant select on coge.* to coge_web;
  flush privileges;
  flush privileges;


Note: The CoGe web-user needs edit/insert permission on some tables. Here is a snapshot of what these are:
Note: The CoGe web-user needs edit/insert permission on some tables. Here is a snapshot of what these are:


=== Deploy new CoGe Web-site ===
=== Deploy the Web Site ===


Download CoGe repository from GitHub  
Generate a public key and add to your GitHub account


git clone https://github.com/LyonsLab/coge.git
See https://help.github.com/articles/generating-an-ssh-key/


Make directories where files go and give write persmission
Download the CoGe repository
 
git clone --recursive https://github.com/LyonsLab/coge.git
 
Run setup script to make required subdirectories


  cd coge/web
  cd coge/web
Line 103: Line 108:


The /etc/apache2/sites-available/default.conf should look like this:<br>  
The /etc/apache2/sites-available/default.conf should look like this:<br>  
<pre>&lt;VirtualHost *&gt;
<pre><VirtualHost *>
ServerAdmin webmaster@localhost
ServerAdmin webmasterl@localhost
DocumentRoot /opt/coge/web
DocumentRoot /opt/coge/web
&lt;Files ~ "\.(pl|cgi)$"&gt;
 
<Files *.pl>
     SetHandler perl-script
     SetHandler perl-script
     PerlResponseHandler ModPerl::Registry
     PerlResponseHandler ModPerl::Registry
     Options +ExecCGI
     Options +ExecCGI
     PerlSendHeader On
     PerlSendHeader On
&lt;/Files&gt;
</Files>


&lt;Directory /&gt;
<Directory />
Options FollowSymLinks
Options FollowSymLinks
AllowOverride None
AllowOverride None
&lt;/Directory&gt;
</Directory>
&lt;Directory /opt/coge/web&gt;
 
PerlSetEnv COGE_HOME "/opt/coge/web"
Alias /gobe/ /opt/coge/web/gobe/
Options Indexes FollowSymLinks MultiViews
<Directory /opt/coge/web/gobe/>
AllowOverride None
Options +FollowSymLinks +ExecCGI
Order allow,deny
AddHandler wsgi-script .py
allow from all
</Directory>
# This directive allows us to have apache2's default start page
 
                # in /apache2-default/, but still have / go to the right place
<Directory /opt/coge>
                #RedirectMatch ^/$ /apache2-default/
Options Includes ExecCGI FollowSymLinks
&lt;/Directory&gt;
    AllowOverride All
    SetEnv COGE_HOME "/opt/coge/"
&lt;Directory /opt/coge/web/services/&gt;
    Order allow,deny
    Allow from all
</Directory>
 
<Directory /opt/coge/web/services/>
         Options +FollowSymLinks +ExecCGI
         Options +FollowSymLinks +ExecCGI
         AddHandler wsgi-script .py
         AddHandler wsgi-script .py
    &lt;/Directory&gt;
</Directory>


    &lt;Directory /opt/coge/web/services/JBrowse/JBrowse_TrackContent_WS/&gt;
<Directory /opt/coge/web/services/JBrowse/JBrowse_TrackContent_WS/>
         Options +FollowSymLinks +ExecCGI
         Options +FollowSymLinks +ExecCGI
         AddHandler wsgi-script .py
         AddHandler wsgi-script .py
    &lt;/Directory&gt;
  </Directory>
    Alias /CoGe/jex /opt/coge/web/services/jex.py
ScriptAliasMatch (?i)^/coge/jex(.*) /opt/coge/web/services/jex.py/$1
    Alias /CoGe/services/JBrowse/track /opt/coge/web/services/JBrowse/J$
        AliasMatch (?i)^/coge(.*) /opt/coge/web/$1
    Alias /CoGe "/opt/coge/web"
        ProxyPass /coge/api/v1/ http://localhost:3303/
 
        ProxyPassReverse /coge/api/v1/ http://localhost:3303/
Alias /gobe/ /opt/coge/web/gobe/
&lt;Directory "/opt/coge/web/gobe/"&gt;
Options +FollowSymLinks +ExecCGI
AddHandler wsgi-script .py
&lt;/Directory&gt;
ScriptALias /api/v1/jbrowse/ /opt/coge/web/services/service.pl
Alias /services/JBrowse/track /opt/coge/web/services/JBrowse/JBrowse_TrackContent_WS/source.py
ScriptAlias /api/v1/ /opt/coge/web/services/mojolicious/
ScriptAlias /cgi-bin/ /opt/coge/web/
&lt;Directory "/opt/coge/web"&gt;
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
AddHandler cgi-script .pl
Order allow,deny
Allow from all
&lt;/Directory&gt;
 
wsgiScriptAlias /jex /opt/coge/web/services/jex.py


ErrorLog /var/log/apache2/error.log
ErrorLog /var/log/apache2/error.log


# Possible values include: debug, info, notice, warn, error, crit,
# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
# alert, emerg.
LogLevel warn
LogLevel warn


CustomLog /var/log/apache2/access.log combined
CustomLog /var/log/apache2/access.log combined
ServerSignature On
ServerSignature On
</VirtualHost>


    Alias /doc/ "/usr/share/doc/"
</pre>
    &lt;Directory "/usr/share/doc/"&gt;
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    &lt;/Directory&gt;


&lt;/VirtualHost&gt;
Enable Required Apache Modules
sudo a2enmod rewrite headers proxy proxy_http expires perl ssl
and reset Apache


</pre>
Configure coge.conf file  
Configure coge.conf file, replacing XXX's with your own information. (Change paths as necessary; this template is configured for having the Coge directory in the path: /opt/coge)


<br>  
Replacing XXXX's with your own information. (Change paths as necessary; this template is configured for having the Coge directory in the path: /opt/coge)<br>  


  ##This is a configuration file for CoGe.
  #
# CoGe Configuration File
#
   
   
  #database configuration
  #database configuration
DB mysql
DBNAME XXXX
DBHOST localhost
DBPORT XXXX
DBUSER XXXX
DBPASS XXXX
#basic auth name and password
AUTHNAME XXXX
AUTHPASS  XXXX
# DE public key for JWT in resources/
DE_PUBLIC_KEY DE_rsa.pub
   
   
 
  #web cookie name
  DBNAME   coge
DBHOST   localhost
DBPORT   3307
DBUSER   coge
DBPASS   XXXXXXX
 
#CAS authentication for webservices
 
CAS_URL https://auth.iplantcollaborative.org/cas
 
#basic auth name and password
 
AUTHNAME XXXXXX
AUTHPASS XXXXXX
 
#web cookie name
 
  COOKIE_NAME cogec
  COOKIE_NAME cogec
#support email address
SUPPORT_EMAIL XXXX
#data dir for coge's programs
DATADIR /storage/coge/data/
#cache dir
CACHEDIR /scratch/coge/cache
#dir for pair-wise whole genome comparisons (e.g. SynMap)
DIAGSDIR /opt/apache2/coge/web/data/diags/
#dir for popgen analysis results
POPGENDIR /storage/coge/data/popgen/
#fasta dir
FASTADIR /opt/apache2/coge/web/data/fasta/
#sequence dir
SEQDIR /storage/coge/data/genomic_sequence/
#experiment dir
EXPDIR /storage/coge/data/experiments/
#temp dir for coge
TEMPDIR /opt/apache2/coge/web/tmp/
#secure temp dir
SECTEMPDIR /scratch/coge/tmp/
# IRODS dir
IRODSDIR /iplant/home/<USER>/coge_data
IRODSSHARED /iplant/home/shared
IRODSENV /opt/apache/coge/irodsEnv
#Base URL for web-site
URL /coge/
API_URL /api/v1/
#URL for temp directory
TEMPURL /coge/tmp/
#blast style scoring matrix dirs
BLASTMATRIX /storage/coge/data/blast/matrix/
#blastable DB
BLASTDB /scratch/coge/cache/blast/db/
#lastable DB
LASTDB /scratch/coge/cache/last/db/
#directory for bed files
BEDDIR /opt/apache2/coge/web/data/bed/
#WIKI URL
WIKI_URL https://genomevolution.org/wiki/index.php
#servername for links
#SERVER https://genomevolution.org/coge/
SERVER http://10.140.65.127/coge/
#CAS URL
CAS_URL https://auth.iplantcollaborative.org/cas4
USER_API_URL https://agave.iplantc.org:443/profiles/v2
MOJOLICIOUS_PORT 3303
# Job Engine Server
JOBSERVER localhost
# Job Engine Port
JOBPORT 5151
#directory for caching genome browser images
IMAGE_CACHE /opt/apache2/coge/web/data/image_cache/
#maximum number of processor to use for multi-CPU systems
MAX_PROC 44
COGE_BLAST_MAX_PROC 8
#True Type Font
FONT /usr/local/fonts/arial.ttf
#various programs
BL2SEQ                /usr/local/bin/legacy_blast.pl bl2seq
BLAST                  /usr/local/bin/legacy_blast.pl blastall
MULTI_LASTZ            /opt/apache2/coge/bin/blastz_wrapper/blastz.py
LAST_PATH              /opt/apache2/coge/bin/last_wrapper/
MULTI_LAST            /opt/apache2/coge/bin/last_wrapper/last.py
LAGAN                  /opt/apache2/coge/bin/lagan-64bit/lagan.pl
LAGANDIR              /opt/apache2/coge/bin/lagan-64bit/
CHAOS                  /opt/apache2/coge/bin/lagan-64bit/chaos
GENOMETHREADER        /opt/apache2/coge/bin/gth
DIALIGN                /opt/apache2/coge/bin/dialign2_dir/dialign2-2_coge
DIALIGN2       /opt/apache2/coge/bin/dialign2_dir/dialign2-2_coge
DIALIGN2_DIR       /opt/apache2/coge/bin/dialign2_dir/
HISTOGRAM       /opt/apache2/coge/bin/histogram.pl
KS_HISTOGRAM       /opt/apache2/coge/bin/ks_histogram.pl
TANDEM_FINDER       /opt/apache2/coge/bin/dagchainer/tandems.py
DAGCHAINER       /opt/apache2/coge/bin/dagchainer_bp/dag_chainer.py
EVALUE_ADJUST       /opt/apache2/coge/bin/dagchainer_bp/dagtools/evalue_adjust.py
FIND_NEARBY       /opt/apache2/coge/bin/dagchainer_bp/dagtools/find_nearby.py
QUOTA_ALIGN       /opt/apache2/coge/bin/quota-alignment/quota_align.py
CLUSTER_UTILS       /opt/apache2/coge/bin/quota-alignment/cluster_utils.py
BLAST2RAW       /opt/apache2/coge/bin/quota-alignment/scripts/blast_to_raw.py
SYNTENY_SCORE       /opt/apache2/coge/bin/quota-alignment/scripts/synteny_score.py
CODEML               /opt/apache2/coge/bin/codeml/codeml-coge
CODEMLCTL              /opt/apache2/coge/bin/codeml/codeml.ctl
CONVERT_BLAST          /opt/apache2/coge/bin/convert_long_blast_to_short_blast_names.pl
DATASETGROUP2BED      /opt/apache2/coge/bin/dataset_group_2_bed.pl
#stuff for Mauve and whole genome alignments
MAUVE        /opt/apache2/coge/bin/GenomeAlign/progressiveMauve-muscleMatrix
COGE_MAUVE  /opt/apache2/coge/bin/GenomeAlign/mauve_alignment.pl
MAUVE_MATRIX /opt/apache2/coge/web/data/blast/matrix/nt/Mauve-Matrix-GenomeAlign
# RNA-seq pipelines
PARSE_CUFFLINKS /opt/apache2/coge/scripts/parse_cufflinks.py
# SNP pipelines
PLATYPUS /opt/apache2/coge/bin/Platypus_0.8.1/Platypus.py
GATK    /opt/apache2/coge/bin/GenomeAnalysisTK.jar
PICARD  /opt/apache2/coge/bin/picard-tools-2.4.1/picard.jar
# ChIP-seq pipeline
HOMER_DIR /opt/apache2/coge/bin/Homer
#THIRD PARTY URLS
GENFAMURL http://dev.gohelle.cirad.fr/genfam/?q=content/upload


#support email address
=== Install Perl Modules  ===


  SUPPORT_EMAIL XXXXXX
*Install cpanminus
  sudo cpan install App::cpanminus


#basedir for coge
*Install third-party modules required by CoGe
cat modules.txt | xargs sudo cpanm


  COGEDIR     /opt/coge/web/
*Manually install Bio::DB::Sam (won't install easily through CPAN, see http://cpansearch.perl.org/src/LDS/Bio-SamTools-1.41/README)
  wget http://search.cpan.org/~lds/Bio-SamTools/
sudo perl INSTALL.pl


#bin dir for coge's programs
*Install CoGe-specific modules


  BINDIR     /opt/coge/web/bin/
  ./make_perl.sh


#scripts dir for coge's programs
* After installing modules, reset the Apache webserver
sudo service apache2 restart


SCRIPTDIR /opt/coge/scripts
=== Install Python Modules ===


RESOURCESDIR /opt/coge/resources
sudo pip install pyzmq matplotlib numpy seaborn natsort requests scipy sklearn


#data dir for coge's programs
=== Install R Modules ===
Note: R version 3.3.0 or higher is required.


  DATADIR     /storage/coge/data/
  sudo R
install.packages("dplyr")
install.packages("useful")
install.packages("gridExtra")


#cache dir
=== Install Javascript dependencies  ===


  CACHEDIR /storage/coge/data/cache/
* Install javascript dependencies
  sudo ln -s /usr/bin/nodejs /usr/bin/node
sudo npm install -g bower
bower install


#dir for pair-wise whole genome comparisons (e.g. SynMap)
=== Install Third-Party Bioinformatics Tools ===


DIAGSDIR     /opt/coge/web/data/diags/
Download the programs listed below and follow the installation instructions on their respective websites.


#fasta dir
Most programs can be installed with the following commands (but check the documentation for each program):
./configure --prefix=/usr/local/
make
sudo make install


  FASTADIR     /opt/coge/web/data/fasta/
* SCIP: http://scip.zib.de/ (for SynMap Syntenic Depth)
* GSNAP/GMAP: http://research-pub.gene.com/gmap/
* FastBit: https://sdm.lbl.gov/fastbit/
* Clustalw: http://www.clustal.org/clustal2/
* GenomeThreader: http://genomethreader.org/
* Bowtie: http://bowtie-bio.sourceforge.net/bowtie2/index.shtml
* TopHat: http://ccb.jhu.edu/software/tophat/index.shtml
* HISAT2:  https://ccb.jhu.edu/software/hisat2/index.shtml
* Cufflinks: http://cole-trapnell-lab.github.io/cufflinks/
* Nwalign: https://pypi.python.org/pypi/nwalign/?
* Cutadapt: http://cutadapt.readthedocs.io/en/stable/installation.html ... sudo su ; sudo pip install cutadapt
* TrimGalore: http://www.bioinformatics.babraham.ac.uk/projects/trim_galore/
* Trimmomatic: http://www.usadellab.org/cms/?page=trimmomatic
* Picard:  http://broadinstitute.github.io/picard/ ... requires Java 8
* Platypus:  http://www.well.ox.ac.uk/platypus
* HSTlib (required by Platypus):  http://www.htslib.org/download/
* Lastz: download the tarball http://www.bx.psu.edu/~rsharris/lastz/  then edit the src/Makefile and remove the word -Werror from line 31. Then run make and make install.
* Last aligner (v731 or greater is required):  http://last.cbrc.jp/
* VCFTools:  https://github.com/vcftools/vcftools
* Vcfutils:  https://github.com/lh3/samtools/blob/master/bcftools/vcfutils.pl
* EMBOSS (sizeseq program):  http://emboss.sourceforge.net/  ... Run "sudo ldconfig" after "make install"
* iCommands: https://github.com/irods/irods-legacy (OPTIONAL: only required if CyVerse authentication and Data Store services are available; note that the legacy version is required, not the latest)
* Bismark:  http://www.bioinformatics.babraham.ac.uk/projects/bismark/
* BWAmeth: https://github.com/brentp/bwa-meth
* BWA: https://sourceforge.net/projects/bio-bwa/
* PileOMeth: https://github.com/dpryan79/PileOMeth
* Homer:  http://homer.salk.edu/homer/ ... sudo perl ./configureHomer.pl -install homer
* Blat (required by Homer): https://genome.ucsc.edu/FAQ/FAQblat.html
* bigWigToWig: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/bigWigToWig
* BedTools:  http://bedtools.readthedocs.io/en/latest/index.html
* SRA Toolkit:  https://github.com/ncbi/sra-tools/wiki/Downloads


#sequence dir
=== Install Third-Party Fonts ===


  SEQDIR     /storage/coge/data/genomic_sequence/
Download from here: https://www.microsoft.com/typography/fonts/font.aspx?FMID=1705
#SEQDIR   /opt/tmp/data/


#experiment dir
And copy to /usr/local/fonts/arial.ttf (or whatever path you set in the coge.conf config file under FONT)


  EXPDIR  /storage/coge/data/experiments/
=== Install blast matrices ===
  cd /storage/coge/data/blast
git clone https://github.com/LyonsLab/blast-matrix.git
mv blast-matrix matrix


#TMPL dir for CoGe's web page templates
=== Install JBrowse  ===


TMPLDIR     /opt/coge/web/tmpl/
Copy from existing CoGe installation if one exists.
Otherwise, download and install the JBrowse package from http://jbrowse.org/install/  


#temp dir for coge
unzip JBrowse-1.11.4-dev.zip
mv JBrowse-1.11.4 /coge/web/js/jbrowse


TEMPDIR   /opt/coge/web/tmp/
=== Install CCTools ===
* Download a stable release that is 6.2.11 or greater from http://ccl.cse.nd.edu/software/downloadfiles.php


#secure temp dir
* Extract the file (this example is using version 6.2.11 which may differ from the version downloaded)
tar xzvf cctools-<version>-source.tar.gz


  SECTEMPDIR /storage/coge/tmp/
* Compile and install
  cd cctools-6.2.11-source
./configure --prefix /usr/local
make
sudo make install


#IRODS dir
* Add the following upstart scripts for the <i>work_queue_pool</i> and <i>catalog_server</i> to <tt>/etc/init</tt>


IRODSDIR        /iplant/home/&lt;USER&gt;/coge_data
By default the pool directory for work_queue will be in /storage/work_queue adjust the directory as needed.
IRODSENV        /opt/coge/web/irodsEnv
<pre>
# /etc/init/.conf


#Base URL for web-server
description "The cctools work queue pool"


URL     /
start on (local-filesystems and net-device-up IFACE=eth0)


#URL for temp directory
stop on shutdown


TEMPURL     /tmp/
respawn limit 30 60


#blast style scoring matrix dirs
pre-start script
    POOL_DIR=/storage/work_queue
    LOG_FILE=$POOL_DIR/logs/work_queue_pool.log


#BLASTMATRIX    /opt/coge/web/data/blast/matrix/
    # Add the pool directory and set ownership
BLASTMATRIX    /storage/coge/data/blast/matrix/
    if ! [ -d "$WORK_DIR" ]; then
        mkdir -p $POOL_DIR/workers
        mkdir -p $POOL_DIR/logs
        chown -R www-data:www-data $POOL_DIR
    fi


#blastable DB
    # Remove the pidfile if it exists
    rm -f $POOL_DIR/work_queue_pool.pid


#BLASTDB    /opt/coge/web/data/blast/db/
    # Archive old log and timestamp the value
  BLASTDB    /storage/coge/data/blast/db/
    if [ -f "$LOG_FILE" ]; then
        TIMESTAMP=$(date +"%Y-%m-%d.%H.%m.%S")
        mv -f $LOG_FILE "$LOGFILE.$TIMESTAMP"
    fi
end script


#lastable DB
script
    POOL_DIR=/storage/work_queue
    LOG_FILE=$POOL_DIR/logs/work_queue_pool.log
    WORK_QUEUE_FACTORY=$(which work_queue_factory)


#LASTDB    /home/franka1/coge/web/data/last/db/
    export CATALOG_HOST=localhost
LASTDB    /storage/coge/data/last/db/
    export CATALOG_PORT=1024


#directory for bed files
    exec start-stop-daemon -c www-data -g www-data -d $POOL_DIR --start \
        -p $POOL_DIR/work_queue_pool.pid --exec $WORK_QUEUE_FACTORY \
        -- -T local -M coge-main -d all -o $LOG_FILE -w 10 \
        -S $POOL_DIR -E "--workdir=$POOL_DIR/workers"
end script
</pre>


BEDDIR   /opt/coge/web/data/bed/
<pre>
# /etc/init/.conf


#servername for links
description "The cctools catalog server"
author "Evan Briones"


SERVER    http://XXXXXX/
start on (local-filesystems and net-device-up IFACE=eth0)


#Job Engine Server
stop on shutdown


JOBSERVER localhost
respawn limit 30 60


#Job Engine Port
script
    exec catalog_server -p 1024 -l 100 -T 3
end script
</pre>


  JOBPORT 5151
* Start the catalog server and work_queue_pool
  sudo start work_queue_pool
sudo start catalog_server


#directory for caching genome browser images
=== Install the Job Engine (Yerba) ===
Download and install the latest Yerba package from&nbsp;https://github.com/LyonsLab/Yerba/archive/v0.3.4.tar.gz


IMAGE_CACHE /opt/coge/web/data/image_cache/
For more specific details on Yerba visit https://github.com/LyonsLab/Yerba/


#maximum number of processor to use for multi-CPU systems
The default installation path for Yerba will be in /opt/Yerba. If another path is chosen update the configuration files to match.


MAX_PROC 32
* Copy and the configuration file to /etc/yerba/yerba.cfg
COGE_BLAST_MAX_PROC 8
<pre>
[DEFAULT]
debug = True
access-log = /opt/Yerba/log/access.log
yerba-log = /opt/Yerba/log/yerbad.log


#True Type Font
[yerba-log]
logging = /etc/yerba/logging.conf


FONT /usr/share/fonts/truetype/msttcorefonts/arial.ttf
[access-log]
logging = /etc/yerba/access.conf


#SynMap workflow tools
[yerba]
port = 5151
level = DEBUG


KSCALC /opt/coge/web/bin/SynMap/kscalc.pl
[workqueue]
GEN_FASTA /opt/coge/web/bin/SynMap/generate_fasta.pl
catalog_server = localhost
RUN_ALIGNMENT /opt/coge/web/bin/SynMap/quota_align_merge.pl
catalog_port = 1024
RUN_COVERAGE /opt/coge/web/bin/SynMap/quota_align_coverage.pl
project = coge-main
PROCESS_DUPS /opt/coge/web/bin/SynMap/process_dups.pl
log = /var/log/workqueue.log
GEVO_LINKS /opt/coge/web/bin/SynMap/gevo_links.pl
port = -1
DOTPLOT_DOTS /opt/coge/web/bin/dotplot_dots.pl
password = /etc/yerba/workqueue_pass
debug = True


#various programs
[db]
path = /opt/Yerba/workflows.db
start_index = 100
</pre>


BL2SEQ /usr/local/bin/legacy_blast.pl bl2seq
* Copy the upstart file to /etc/upstart/yerba.conf
BLASTZ /usr/local/bin/blastz
<pre>
LASTZ /usr/local/bin/lastz
# /etc/init/yerba.conf
MULTI_LASTZ /opt/coge/web/bin/blastz_wrapper/blastz.py
LAST_PATH /opt/coge/web/bin/last_wrapper/
MULTI_LAST /opt/coge/web/bin/last_wrapper/last.py
#BLAST 2.2.23+
BLAST /usr/local/bin/legacy_blast.pl blastall
TBLASTN  /usr/local/bin/tblastn
BLASTN /usr/local/bin/blastn
BLASTP /usr/local/bin/blastp
TBLASTX /usr/local/bin/tblastx
FASTBIT_LOAD  /usr/local/bin/ardea
FASTBIT_QUERY /usr/local/bin/ibis
SAMTOOLS /usr/bin/samtools
RAZIP /usr/local/bin/razip


###Formatdb needs to be updated to makeblastdb
description "Yerba server daemon"
author "Evan Briones"


FORMATDB /usr/bin/formatdb
start on (local-filesystems and net-device-up IFACE=eth0)
LAGAN /opt/coge/web/bin/lagan-64bit/lagan.pl
LAGANDIR /opt/coge/web/bin/lagan-64bit/
CHAOS /opt/coge/web/bin/lagan-64bit/chaos
GENOMETHREADER /opt/coge/web/bin/gth
DIALIGN       /opt/coge/web/bin/dialign2_dir/dialign2-2_coge
DIALIGN2       /opt/coge/web/bin/dialign2_dir/dialign2-2_coge
DIALIGN2_DIR       /opt/coge/web/bin/dialign2_dir/
HISTOGRAM       /opt/coge/web/bin/histogram.pl
KS_HISTOGRAM       /opt/coge/web/bin/ks_histogram.pl
PYTHON       /usr/bin/python
PYTHON26   /usr/bin/python
DAG_TOOL       /opt/coge/web/bin/SynMap/dag_tools.py
BLAST2BED       /opt/coge/web/bin/SynMap/blast2bed.pl
TANDEM_FINDER       /opt/coge/web/bin/dagchainer/tandems.py
DAGCHAINER       /opt/coge/web/bin/dagchainer_bp/dag_chainer.py
EVALUE_ADJUST       /opt/coge/web/bin/dagchainer_bp/dagtools/evalue_adjust.py
FIND_NEARBY       /opt/coge/web/bin/dagchainer_bp/dagtools/find_nearby.py
QUOTA_ALIGN       /opt/coge/web/bin/quota-alignment/quota_align.py
CLUSTER_UTILS       /opt/coge/web/bin/quota-alignment/cluster_utils.py
BLAST2RAW       /opt/coge/web/bin/quota-alignment/scripts/blast_to_raw.py
SYNTENY_SCORE       /opt/coge/web/bin/quota-alignment/scripts/synteny_score.py
DOTPLOT       /opt/coge/web/bin/dotplot.pl
SVG_DOTPLOT       /opt/coge/web/bin/SynMap/dotplot.py
NWALIGN       /usr/local/bin/nwalign
CODEML       /opt/coge/web/bin/codeml/codeml-coge
CODEMLCTL      /opt/coge/web/bin/codeml/codeml.ctl
CONVERT_BLAST  /opt/coge/web/bin/convert_long_blast_to_short_blast_names.pl
DATASETGROUP2BED /opt/coge/web/bin/dataset_group_2_bed.pl
ARAGORN /usr/local/bin/aragorn
CLUSTALW /usr/local/bin/clustalw2
GZIP /bin/gzip
GUNZIP /bin/gunzip
TAR /bin/tar


#MotifView
stop on shutdown


MOTIF_FILE /opt/coge/web/bin/MotifView/motif_hash_dump
respawn


#stuff for Mauve and whole genome alignments
pre-start script
    LOG_DIR=/opt/Yerba/log
    LOG_FILE=$LOG_DIR/debug.log
    [ -d "$LOG_DIR" ] || mkdir -m777 -p $LOG_DIR
#   [ -f "$LOG_FILE" ] || rm -f $LOG_FILE
end script


MAUVE /opt/coge/web/bin/GenomeAlign/progressiveMauve-muscleMatrix
script
COGE_MAUVE /opt/coge/web/bin/GenomeAlign/mauve_alignment.pl
    export YERBA_ROOT=/opt/Yerba
MAUVE_MATRIX /opt/coge/web/data/blast/matrix/nt/Mauve-Matrix-GenomeAlign
    export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$YERBA_ROOT"
#newicktops is part of njplot package
    exec start-stop-daemon -c www-data -g www-data --start \
NEWICKTOPS /usr/bin/newicktops
        --iosched real-time --nicelevel -19 \
#convert is from ImageMagick
        --exec $YERBA_ROOT/bin/yerbad -- >> $YERBA_ROOT/log/debug.log 2>&1
CONVERT /usr/bin/convert
end script
#from graphviz
DOT
NEATO


<br>
post-start script
 
    echo Restart on: `hostname -A` | mail -s "UPSTART: Yerba was started" coge.genome@gmail.com
CUTADAPT /usr/local/bin/cutadapt
end script
GSNAP /usr/local/bin/gsnap
</pre>
CUFFLINKS /usr/local/bin/cufflinks
PARSE_CUFFLINKS /opt/coge/scripts/parse_cufflinks.py
GMAP_BUILD /usr/local/bin/gmap_build
BOWTIE_BUILD /usr/local/bin/bowtie2-build
TOPHAT /usr/local/bin/tophat
 
#THIRD PARTY URLS
 
GENFAMURL http://dev.gohelle.cirad.fr/genfam/?q=content/upload
GRIMMURL http://grimm.ucsd.edu/cgi-bin/grimm.cgi#report
QTELLER_URL http://geco.iplantc.org/qTeller
 
=== Install Perl Modules  ===
 
*Install CoGe modules /modules directory of CoGe install path


  perl Makefile.PL lib=/usr/local/lib/perl; make install
* Initialize and start the job engine
 
  /opt/Yerba/bin/yerbad --setup
*Install non-standard Perl modules
  sudo chown www-data:www-data /opt/Yerba/workflows.db
 
  sudo start yerba
AuthCAS
DBIx::Class
DBIx::Class::Schema
DBD::SQLite
Data::UUID
Data::GUID
  XML::SAX::Expat (requires: sudo apt-get install expat libexpat1-dev)
XML::Parser
XML::Simple
IPC::System::Simple
CGI::Application::Dispatch
CGI::Log
GD::Graph::bars
Moose
ZMQ::LibZMQ3
File::Slurp
DateTime
URI::Split
 
=== Install JBrowse  ===
 
Download and install the JBrowse package from http://jbrowse.org/install/  
 
cd js
unzip JBrowse-1.9.7.zip
  mv JBrowse-1.9.7 jbrowse
 
=== Populate with test data  ===
 
scripts/replicate_genome_between_coge_installations.pl -dsgid 11022 -u1 coge -p1 XXXXXXX -db1 coge -u2 oryza_coge -p2 XXXXXXX -db2 oryza_coge -sd /opt/apache/oryz_coge/data/genomic_sequence


== Troubleshooting  ==
== Troubleshooting  ==
Line 468: Line 586:


*Apache configuration for gobe  
*Apache configuration for gobe  
*Make sure the Python Web module is installed:  sudo aptitude install python-webpy
*Check to see if paths hard-coded into gobe/flash/service.wsgi need to be updated  
*Check to see if paths hard-coded into gobe/flash/service.wsgi need to be updated  
**NOTE: Not sure if this is required
**NOTE: Not sure if this is required
=== Working on an Atmosphere Virtual Machine  ===
Click [https://genomevolution.org/wiki/index.php/Coge_VM here] for instructions on dealing with issues that occur specifically with Atmosphere Virtual machines.

Latest revision as of 18:07, 15 May 2023

Installing CoGe on Ubuntu

Note: these instructions were last updated and verified on June 3rd, 2016 by mdb.

Initial Dependencies

Run the following command: 

sudo apt-get -y install {package}

where {package} is each of the following: 

apache2
aragorn
blast2
build-essential
checkinstall
expat
gcc-multilib
git
graphviz
imagemagick
libdb-dev
libgd2-xpm-dev
libperl-dev
libgd-gd2-perl
libconfig-yaml-perl
libssl-dev
libzmq3-dev
mysql-server
ncbi-blast+
ncbi-blast+-legacy
njplot
phpmyadmin
python-dev
python-numpy
python-software-properties
samtools
swig
sqlite3
ttf-mscorefonts-installer
ubuntu-dev-tools
libapache-asp-perl
libapache2-mod-perl2
libapache2-mod-wsgi
python-pip
r-cran-plyr
r-cran-reshape2
r-cran-ggplot2
nodejs
npm
libboost-all-dev  (for TopHat)
python-glpk
glpk-utils
libgmp3-dev
zimpl

Create MySQL database

Dump CoGe database schema (if using existing CoGe installation, otherwise see schema file below). 

mysqldump -d -h localhost -u root -pXXXXXXX coge | sed 's/AUTO_INCREMENT=[0-9]*\b//' > coge_mysql_schema.sql

Create new CoGe Database 

create database coge

Initialize new coge database 

mysql -u root -pXXXXXXXX coge < coge_mysql_schema.sql

Populate a few entries in the feature_type table 

mysql -u root -pXXXXXXXXX coge < coge_feature_types.sql

Create new MySQL user for the CoGe database 

use mysql;
create user 'coge'@'localhost' IDENTIFIED BY 'XXXXXX';
grant all privileges on coge.* to coge;
flush privileges;

Note: The CoGe web-user needs edit/insert permission on some tables. Here is a snapshot of what these are:

Deploy the Web Site

Generate a public key and add to your GitHub account

See https://help.github.com/articles/generating-an-ssh-key/

Download the CoGe repository 

git clone --recursive https://github.com/LyonsLab/coge.git

Run setup script to make required subdirectories 

cd coge/web
./setup.sh

Configure apache

The /etc/apache2/sites-available/default.conf should look like this:

<VirtualHost *>
	ServerAdmin webmasterl@localhost
	DocumentRoot /opt/coge/web

	<Files *.pl>
    		SetHandler perl-script
    		PerlResponseHandler ModPerl::Registry
    		Options +ExecCGI
    		PerlSendHeader On
	</Files>

	<Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>

	Alias /gobe/ /opt/coge/web/gobe/
	<Directory /opt/coge/web/gobe/>
		Options +FollowSymLinks +ExecCGI
		AddHandler wsgi-script .py
	</Directory>

	<Directory /opt/coge>
		Options Includes ExecCGI FollowSymLinks
    		AllowOverride All
    		SetEnv COGE_HOME "/opt/coge/"
    		Order allow,deny
    		Allow from all
	</Directory>

	<Directory /opt/coge/web/services/>
        	Options +FollowSymLinks +ExecCGI
        	AddHandler wsgi-script .py
 	</Directory>

 	<Directory /opt/coge/web/services/JBrowse/JBrowse_TrackContent_WS/>
        	Options +FollowSymLinks +ExecCGI
        	AddHandler wsgi-script .py
  	</Directory>
	ScriptAliasMatch (?i)^/coge/jex(.*) /opt/coge/web/services/jex.py/$1
        AliasMatch (?i)^/coge(.*) /opt/coge/web/$1
        ProxyPass /coge/api/v1/ http://localhost:3303/
        ProxyPassReverse /coge/api/v1/ http://localhost:3303/

	ErrorLog /var/log/apache2/error.log

	# Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
	LogLevel warn

	CustomLog /var/log/apache2/access.log combined
	ServerSignature On
</VirtualHost>

Enable Required Apache Modules 

sudo a2enmod rewrite headers proxy proxy_http expires perl ssl

and reset Apache

Configure coge.conf file

Replacing XXXX's with your own information. (Change paths as necessary; this template is configured for having the Coge directory in the path: /opt/coge)

#
# CoGe Configuration File
#

#database configuration
DB mysql
DBNAME XXXX
DBHOST localhost
DBPORT XXXX
DBUSER XXXX
DBPASS XXXX

#basic auth name and password
AUTHNAME XXXX
AUTHPASS  XXXX

# DE public key for JWT in resources/
DE_PUBLIC_KEY DE_rsa.pub

#web cookie name
COOKIE_NAME cogec

#support email address
SUPPORT_EMAIL XXXX

#data dir for coge's programs
DATADIR	/storage/coge/data/

#cache dir
CACHEDIR /scratch/coge/cache

#dir for pair-wise whole genome comparisons (e.g. SynMap)
DIAGSDIR /opt/apache2/coge/web/data/diags/

#dir for popgen analysis results
POPGENDIR /storage/coge/data/popgen/

#fasta dir
FASTADIR /opt/apache2/coge/web/data/fasta/

#sequence dir
SEQDIR /storage/coge/data/genomic_sequence/

#experiment dir
EXPDIR /storage/coge/data/experiments/

#temp dir for coge
TEMPDIR /opt/apache2/coge/web/tmp/

#secure temp dir
SECTEMPDIR /scratch/coge/tmp/

# IRODS dir
IRODSDIR /iplant/home/<USER>/coge_data
IRODSSHARED /iplant/home/shared
IRODSENV /opt/apache/coge/irodsEnv

#Base URL for web-site
URL /coge/
API_URL /api/v1/

#URL for temp directory
TEMPURL /coge/tmp/

#blast style scoring matrix dirs
BLASTMATRIX /storage/coge/data/blast/matrix/

#blastable DB
BLASTDB /scratch/coge/cache/blast/db/

#lastable DB
LASTDB /scratch/coge/cache/last/db/

#directory for bed files
BEDDIR /opt/apache2/coge/web/data/bed/

#WIKI URL
WIKI_URL https://genomevolution.org/wiki/index.php

#servername for links
#SERVER https://genomevolution.org/coge/
SERVER http://10.140.65.127/coge/

#CAS URL
CAS_URL https://auth.iplantcollaborative.org/cas4
USER_API_URL https://agave.iplantc.org:443/profiles/v2
MOJOLICIOUS_PORT 3303

# Job Engine Server
JOBSERVER localhost

# Job Engine Port
JOBPORT 5151

#directory for caching genome browser images
IMAGE_CACHE /opt/apache2/coge/web/data/image_cache/

#maximum number of processor to use for multi-CPU systems
MAX_PROC 44
COGE_BLAST_MAX_PROC 8

#True Type Font
FONT /usr/local/fonts/arial.ttf

#various programs
BL2SEQ                 /usr/local/bin/legacy_blast.pl bl2seq
BLAST                  /usr/local/bin/legacy_blast.pl blastall
MULTI_LASTZ            /opt/apache2/coge/bin/blastz_wrapper/blastz.py
LAST_PATH              /opt/apache2/coge/bin/last_wrapper/
MULTI_LAST             /opt/apache2/coge/bin/last_wrapper/last.py
LAGAN                  /opt/apache2/coge/bin/lagan-64bit/lagan.pl
LAGANDIR               /opt/apache2/coge/bin/lagan-64bit/
CHAOS                  /opt/apache2/coge/bin/lagan-64bit/chaos
GENOMETHREADER         /opt/apache2/coge/bin/gth
DIALIGN                /opt/apache2/coge/bin/dialign2_dir/dialign2-2_coge
DIALIGN2	       /opt/apache2/coge/bin/dialign2_dir/dialign2-2_coge
DIALIGN2_DIR	       /opt/apache2/coge/bin/dialign2_dir/
HISTOGRAM	       /opt/apache2/coge/bin/histogram.pl
KS_HISTOGRAM	       /opt/apache2/coge/bin/ks_histogram.pl
TANDEM_FINDER	       /opt/apache2/coge/bin/dagchainer/tandems.py
DAGCHAINER	       /opt/apache2/coge/bin/dagchainer_bp/dag_chainer.py
EVALUE_ADJUST	       /opt/apache2/coge/bin/dagchainer_bp/dagtools/evalue_adjust.py
FIND_NEARBY	       /opt/apache2/coge/bin/dagchainer_bp/dagtools/find_nearby.py
QUOTA_ALIGN	       /opt/apache2/coge/bin/quota-alignment/quota_align.py
CLUSTER_UTILS	       /opt/apache2/coge/bin/quota-alignment/cluster_utils.py
BLAST2RAW	       /opt/apache2/coge/bin/quota-alignment/scripts/blast_to_raw.py
SYNTENY_SCORE	       /opt/apache2/coge/bin/quota-alignment/scripts/synteny_score.py
CODEML	               /opt/apache2/coge/bin/codeml/codeml-coge
CODEMLCTL              /opt/apache2/coge/bin/codeml/codeml.ctl
CONVERT_BLAST          /opt/apache2/coge/bin/convert_long_blast_to_short_blast_names.pl
DATASETGROUP2BED       /opt/apache2/coge/bin/dataset_group_2_bed.pl

#stuff for Mauve and whole genome alignments
MAUVE        /opt/apache2/coge/bin/GenomeAlign/progressiveMauve-muscleMatrix
COGE_MAUVE   /opt/apache2/coge/bin/GenomeAlign/mauve_alignment.pl
MAUVE_MATRIX /opt/apache2/coge/web/data/blast/matrix/nt/Mauve-Matrix-GenomeAlign

# RNA-seq pipelines
PARSE_CUFFLINKS /opt/apache2/coge/scripts/parse_cufflinks.py

# SNP pipelines
PLATYPUS /opt/apache2/coge/bin/Platypus_0.8.1/Platypus.py
GATK     /opt/apache2/coge/bin/GenomeAnalysisTK.jar
PICARD   /opt/apache2/coge/bin/picard-tools-2.4.1/picard.jar

# ChIP-seq pipeline
HOMER_DIR /opt/apache2/coge/bin/Homer

#THIRD PARTY URLS
GENFAMURL http://dev.gohelle.cirad.fr/genfam/?q=content/upload

Install Perl Modules

  • Install cpanminus
sudo cpan install App::cpanminus
  • Install third-party modules required by CoGe
cat modules.txt | xargs sudo cpanm

wget http://search.cpan.org/~lds/Bio-SamTools/
sudo perl INSTALL.pl
  • Install CoGe-specific modules
./make_perl.sh
  • After installing modules, reset the Apache webserver
sudo service apache2 restart

Install Python Modules

sudo pip install pyzmq matplotlib numpy seaborn natsort requests scipy sklearn

Install R Modules

Note: R version 3.3.0 or higher is required. 

sudo R
install.packages("dplyr")
install.packages("useful")
install.packages("gridExtra")

Install Javascript dependencies

  • Install javascript dependencies
sudo ln -s /usr/bin/nodejs /usr/bin/node
sudo npm install -g bower
bower install

Install Third-Party Bioinformatics Tools

Download the programs listed below and follow the installation instructions on their respective websites.

Most programs can be installed with the following commands (but check the documentation for each program): 

./configure --prefix=/usr/local/
make
sudo make install

Install Third-Party Fonts

Download from here: https://www.microsoft.com/typography/fonts/font.aspx?FMID=1705

And copy to /usr/local/fonts/arial.ttf (or whatever path you set in the coge.conf config file under FONT)

Install blast matrices

cd /storage/coge/data/blast
git clone https://github.com/LyonsLab/blast-matrix.git
mv blast-matrix matrix

Install JBrowse

Copy from existing CoGe installation if one exists. Otherwise, download and install the JBrowse package from http://jbrowse.org/install/ 

unzip JBrowse-1.11.4-dev.zip
mv JBrowse-1.11.4 /coge/web/js/jbrowse

Install CCTools

  • Extract the file (this example is using version 6.2.11 which may differ from the version downloaded)
tar xzvf cctools-<version>-source.tar.gz
  • Compile and install
cd cctools-6.2.11-source
./configure --prefix /usr/local
make
sudo make install
  • Add the following upstart scripts for the work_queue_pool and catalog_server to /etc/init

By default the pool directory for work_queue will be in /storage/work_queue adjust the directory as needed. 

# /etc/init/.conf

description "The cctools work queue pool"

start on (local-filesystems and net-device-up IFACE=eth0)

stop on shutdown

respawn limit 30 60

pre-start script
    POOL_DIR=/storage/work_queue
    LOG_FILE=$POOL_DIR/logs/work_queue_pool.log

    # Add the pool directory and set ownership
    if ! [ -d "$WORK_DIR" ]; then
        mkdir -p $POOL_DIR/workers
        mkdir -p $POOL_DIR/logs
        chown -R www-data:www-data $POOL_DIR
    fi

    # Remove the pidfile if it exists
    rm -f $POOL_DIR/work_queue_pool.pid

    # Archive old log and timestamp the value
    if [ -f "$LOG_FILE" ]; then
        TIMESTAMP=$(date  +"%Y-%m-%d.%H.%m.%S")
        mv -f $LOG_FILE "$LOGFILE.$TIMESTAMP"
    fi
end script

script
    POOL_DIR=/storage/work_queue
    LOG_FILE=$POOL_DIR/logs/work_queue_pool.log
    WORK_QUEUE_FACTORY=$(which work_queue_factory)

    export CATALOG_HOST=localhost
    export CATALOG_PORT=1024

    exec start-stop-daemon -c www-data -g www-data -d $POOL_DIR --start \
        -p $POOL_DIR/work_queue_pool.pid --exec $WORK_QUEUE_FACTORY \
        -- -T local -M coge-main -d all -o $LOG_FILE -w 10 \
        -S $POOL_DIR -E "--workdir=$POOL_DIR/workers"
end script

# /etc/init/.conf

description "The cctools catalog server"
author "Evan Briones"

start on (local-filesystems and net-device-up IFACE=eth0)

stop on shutdown

respawn limit 30 60

script
    exec catalog_server -p 1024 -l 100 -T 3
end script
  • Start the catalog server and work_queue_pool
sudo start work_queue_pool
sudo start catalog_server

Install the Job Engine (Yerba)

Download and install the latest Yerba package from https://github.com/LyonsLab/Yerba/archive/v0.3.4.tar.gz

For more specific details on Yerba visit https://github.com/LyonsLab/Yerba/

The default installation path for Yerba will be in /opt/Yerba. If another path is chosen update the configuration files to match.

  • Copy and the configuration file to /etc/yerba/yerba.cfg
[DEFAULT]
debug = True
access-log = /opt/Yerba/log/access.log
yerba-log = /opt/Yerba/log/yerbad.log

[yerba-log]
logging = /etc/yerba/logging.conf

[access-log]
logging = /etc/yerba/access.conf

[yerba]
port = 5151
level = DEBUG

[workqueue]
catalog_server = localhost
catalog_port = 1024
project = coge-main
log = /var/log/workqueue.log
port = -1
password = /etc/yerba/workqueue_pass
debug = True

[db]
path = /opt/Yerba/workflows.db
start_index = 100
  • Copy the upstart file to /etc/upstart/yerba.conf
# /etc/init/yerba.conf

description "Yerba server daemon"
author "Evan Briones"

start on (local-filesystems and net-device-up IFACE=eth0)

stop on shutdown

respawn

pre-start script
    LOG_DIR=/opt/Yerba/log
    LOG_FILE=$LOG_DIR/debug.log
    [ -d "$LOG_DIR" ] || mkdir -m777 -p $LOG_DIR
#    [ -f "$LOG_FILE" ] || rm -f $LOG_FILE
end script

script
    export YERBA_ROOT=/opt/Yerba
    export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$YERBA_ROOT"
    exec start-stop-daemon -c www-data -g www-data --start \
        --iosched real-time --nicelevel -19 \
        --exec $YERBA_ROOT/bin/yerbad -- >> $YERBA_ROOT/log/debug.log 2>&1
end script

post-start script
    echo Restart on: `hostname -A` | mail -s "UPSTART: Yerba was started" coge.genome@gmail.com
end script
  • Initialize and start the job engine
/opt/Yerba/bin/yerbad --setup
sudo chown www-data:www-data /opt/Yerba/workflows.db
sudo start yerba

Troubleshooting

Visualization in GEvo does not work

This relies on a system known as Gobe. Check the following things:

  • Apache configuration for gobe
  • Make sure the Python Web module is installed: sudo aptitude install python-webpy
  • Check to see if paths hard-coded into gobe/flash/service.wsgi need to be updated
    • NOTE: Not sure if this is required

Working on an Atmosphere Virtual Machine

Click here for instructions on dealing with issues that occur specifically with Atmosphere Virtual machines.

Retrieved from "https://genomevolution.org/wiki/index.php?title=Install_coge&oldid=9865"