CoGepedia:Current events: Difference between revisions

From CoGepedia
Jump to navigation Jump to search
No edit summary
Line 38: Line 38:
Nobody likes to wait for anything, especially when it comes to comparative genomics.  To a large extent, CoGe is designed to allow researchers to quickly go from an idea or hypothesis, to testing it, to visualizing and interpreting the result, to getting a new idea, to testing that . . . so when you have to wait for the spinning double-helix to stop to get your results, you may lose momentum.  Thus, while digging through the code to update the API, I took time to find areas of CoGe that were slow and tried to bump up the performance.
Nobody likes to wait for anything, especially when it comes to comparative genomics.  To a large extent, CoGe is designed to allow researchers to quickly go from an idea or hypothesis, to testing it, to visualizing and interpreting the result, to getting a new idea, to testing that . . . so when you have to wait for the spinning double-helix to stop to get your results, you may lose momentum.  Thus, while digging through the code to update the API, I took time to find areas of CoGe that were slow and tried to bump up the performance.


This fell into four general classes  
This fell into four general classes:
*Server-side parallelization
====Server-side parallelization====
Currently CoGe runs an 8-core server, and the process of parallelization means figuring out areas of a computational program that can be run in parallel and divvied up among the multiple CPUs, or sent out to another server with extra processing power.  While much of CoGe already ran in parallel, there were more places the methodology could be implemented.  However, the problem was finding the places to parallelize that would result in a substantial performance increase for most analyses. 
 
GEvo, CoGe's tools for comparing multiple genomic regions in high-detail, was high on the list for such optimizations.  I had already had all the pair-wise sequence comparisons running in parallel, but the images of the results were being generated in series.  When I timed each stage of a GEvo analysis, I found that in almost all cases, the image generation was the slowest step of the analysis.  Sometimes being an order of magnitude slower that the rest of the analysis combined.  When parallelized, up to eight images would be generated at the same time, which is usually many more genomic regions than are usually analyzed at once.
 
Other places that were parallelized required modifying programs written by other researchers.  For example, Codeml, part of the [[http://abacus.gene.ucl.ac.uk/software/paml.html PAML]] bioinformatics software suite written by Ziheng Yang, is used to calculate synonymous/non-synonymous (Ks, Kn) rate changes between two codon-aligned DNA sequences.  I wanted to integrate Ks and Kn calculations into SynMap, CoGe's tool for generating whole genome syntenic dotplots for identifying syntenic regions.  Unfortunately, Codeml was written to accept input and output files to specify sequences and configuration of the analysis.  This presented major problems for parallelization because of the need to make sure that configuration and sequence files were not accidentally crossed-used.  Also, this caused a decrease in performance due to the time needed to read and write these files to the hard drives.  For this problem, Brent took on the challenge of rewriting Codeml to accept sequences and configuration parameters from the command-line.  Now, Ks calculations can be performed on syntenic regions identified by SynMap, and the resulting dotplot colored based on those values to allow researchers to quickly identify regions of increased divergence, multiple whole genome duplications, and other such phenomena.  (Although, I need to state that it still takes several hours to generate these data for tens of thousands of gene pairs.  But this is a substantial improvement over the days it used to take!)
 
*Client-side javascript rewrites
*Client-side javascript rewrites
As the web-interface to CoGe evolved, we used more and more javascript to make the system behave more like a desktop application.  This degree of user control on the interface was needed when dealing with the large amounts of data CoGe could generate.  CoGeBlast is a prime example of this as this tool lets a researcher blast as many sequences as they want against nearly any number of genomes in CoGe.  The utility in CoGeBlast is the integration of blast-results with underlying genomic information.  However, there is a lot of data you can get, and not everyone is going to want to see all of it, and more importantly, different questions may require seeing different types of data.  The table CoGeBlast has for listing blast hits is interactive, allowing researchers to sort on one or more columns of data, select which columns of data they wish to view, and select genomic regions their sequences hit for further analysis.  The first implementation of the column hiding and showing algorithm worked, but could be very slow when used on a large table of results, or on an older computer.  This has been updated to use more efficient logic and javascript routines, so the amount of time waiting for columns to hide is greatly decreased.
*Preservation of large analyses and data files
*Preservation of large analyses and data files
If you are going to compare two genomes consisting of hundreds of megabases of sequences, changes are you are going to want to look at the results again
*Algorithm improvements
*Algorithm improvements

Revision as of 00:09, 16 July 2009

Version 3 of CoGe is released!

General Announcement

July 14th, 2009

After a 6-month round of development, I am happy to announce that a new version of CoGe is ready for use.   This major release includes:

  • Modification of the CoGe database schema (and the resulting data migration issues)
  • Update of the database API
  • Update of all web programs to new API
  • Performance tuning of web applications
  • Update of web applications' user interface (predominately using jQuery)
  • A grossly under-populated wiki!

Thanks to:

  • Josh Kane
  • Brent Pedersen
  • Dan Hembry
  • Damon Lisch
  • Michael Freeling


--Eric Lyons

Database Schema Change Details

If it wasn't for CoGe's Database, nothing else in the system would work. This is the core of CoGe, and in a relatively simple schema, allows for multiple versions of multiple genomes, from multiple data sources, of multiple organisms to all be stored in the same place. As the number of genomes in CoGe grew, the database scaled very well, easily accommodating over 6000 genomes and 80 Gigabases of sequence. The overall size of the database was around 100GB, with several million entries spread over its tables. While the system could handle this much data, our snapshot and backup system did not. CoGe uses MySQL to drive its database, and every time a table was changed (e.g. add a new genome), all the binary files of the database changed, and our backup system started copying another 100GB of data. As new genomes are added to the system regularly, we quickly ran into problems with our backup server as our couple terabytes of storage were getting filled in days, and we needed backups spanning months just in case something goes really, really bad. The solution was simple -- move the raw genomic sequence data out of the database and into flat-files. This shrank our database size to around 20GB, and made our backup system much happier. Overall, this did not result in a serious rewrite of the API, and had an extra benefit of substantially speeding up the retrieval of genome sizes from the database (essential for some comparative genomic work.) Instead, I have Brent to thank for inspiring the major change in the API.

Brent had the great idea of separating the genomic sequence data from the datasets that specify genomic features and annotations (e.g. genes, proteins, repeat sequences). This would allow use to take a genome, mask it any way we like, and immediately be able to hook up the newly processed genomic sequence to it original annotations. Also, if we (or others) created a custom set of annotations (e.g. conserved non-coding sequences, small peptides, missed annotations) we could easily add those annotations to a pre-existing genome without needing to duplicate data. Also, this allows for the creation of a unified "genome", which is actually a lot more difficult to do from a data-management standpoint when dealing with genomes sequenced by anyone.

While this new schema promised a substantial improvement in how we managed genomic data, implementing it in CoGe was difficult. Since our group, and others, have published datasets and papers with links to CoGe for a particular set of sequences or analyses that sometimes use CoGe's unique database ids, the unique database ids for major data points needed to be faithfully mapped. Also, since this data comes from several places, some that can be retrieved in an automated fashion, others requiring much more manual work, reloading everything from their raw data files would have been painful. Also, this database schema change reflected a break in how CoGe relates two major sets of data -- genomic sequence and features. In order to get this schema to work with the existing suite of tools, the API needed some substantial work, and every web-based and command-line tool in CoGe needed to be updated to reflect these changes.

Fortunately, I had just finished writing and submitting my PhD thesis, and was looking to get back to coding.

Performance Tuning

Nobody likes to wait for anything, especially when it comes to comparative genomics. To a large extent, CoGe is designed to allow researchers to quickly go from an idea or hypothesis, to testing it, to visualizing and interpreting the result, to getting a new idea, to testing that . . . so when you have to wait for the spinning double-helix to stop to get your results, you may lose momentum. Thus, while digging through the code to update the API, I took time to find areas of CoGe that were slow and tried to bump up the performance.

This fell into four general classes:

Server-side parallelization

Currently CoGe runs an 8-core server, and the process of parallelization means figuring out areas of a computational program that can be run in parallel and divvied up among the multiple CPUs, or sent out to another server with extra processing power. While much of CoGe already ran in parallel, there were more places the methodology could be implemented. However, the problem was finding the places to parallelize that would result in a substantial performance increase for most analyses.

GEvo, CoGe's tools for comparing multiple genomic regions in high-detail, was high on the list for such optimizations. I had already had all the pair-wise sequence comparisons running in parallel, but the images of the results were being generated in series. When I timed each stage of a GEvo analysis, I found that in almost all cases, the image generation was the slowest step of the analysis. Sometimes being an order of magnitude slower that the rest of the analysis combined. When parallelized, up to eight images would be generated at the same time, which is usually many more genomic regions than are usually analyzed at once.

Other places that were parallelized required modifying programs written by other researchers. For example, Codeml, part of the [PAML] bioinformatics software suite written by Ziheng Yang, is used to calculate synonymous/non-synonymous (Ks, Kn) rate changes between two codon-aligned DNA sequences. I wanted to integrate Ks and Kn calculations into SynMap, CoGe's tool for generating whole genome syntenic dotplots for identifying syntenic regions. Unfortunately, Codeml was written to accept input and output files to specify sequences and configuration of the analysis. This presented major problems for parallelization because of the need to make sure that configuration and sequence files were not accidentally crossed-used. Also, this caused a decrease in performance due to the time needed to read and write these files to the hard drives. For this problem, Brent took on the challenge of rewriting Codeml to accept sequences and configuration parameters from the command-line. Now, Ks calculations can be performed on syntenic regions identified by SynMap, and the resulting dotplot colored based on those values to allow researchers to quickly identify regions of increased divergence, multiple whole genome duplications, and other such phenomena. (Although, I need to state that it still takes several hours to generate these data for tens of thousands of gene pairs. But this is a substantial improvement over the days it used to take!)

  • Client-side javascript rewrites

As the web-interface to CoGe evolved, we used more and more javascript to make the system behave more like a desktop application. This degree of user control on the interface was needed when dealing with the large amounts of data CoGe could generate. CoGeBlast is a prime example of this as this tool lets a researcher blast as many sequences as they want against nearly any number of genomes in CoGe. The utility in CoGeBlast is the integration of blast-results with underlying genomic information. However, there is a lot of data you can get, and not everyone is going to want to see all of it, and more importantly, different questions may require seeing different types of data. The table CoGeBlast has for listing blast hits is interactive, allowing researchers to sort on one or more columns of data, select which columns of data they wish to view, and select genomic regions their sequences hit for further analysis. The first implementation of the column hiding and showing algorithm worked, but could be very slow when used on a large table of results, or on an older computer. This has been updated to use more efficient logic and javascript routines, so the amount of time waiting for columns to hide is greatly decreased.


  • Preservation of large analyses and data files

If you are going to compare two genomes consisting of hundreds of megabases of sequences, changes are you are going to want to look at the results again

  • Algorithm improvements