How to create your own replica of the SureChEMBL patent-chemistry dataset

Introduction: Why replicate SureChEMBL?

SureChEMBL is a patent chemistry dataset and set of web services that provides a rich source of information to the drug discovery research community. It was previously owned, developed, and sold by Macmillan, but was recently handed over to the European Bioinformatics Institute (EMBL/EBI) and is now free for everyone to use.

SureChEMBL can already be accessed online, so why would a locally hosted replica be needed?

To answer that question, I'll give the reasons provided by a pharmaceutical company who recently commissioned me to develop a SureChEMBL data replication facility:

• 1) Firewall restrictions can be avoided - companies involved in drug discovery are often working with substructures or other related search queries which may lead to highly lucrative discoveries. As such, researchers are often prohibited from using external web services, even secure services such as SureChEMBL, as a risk-mitigation strategy. Downloading data files - e.g. exported patent chemistry data - via standard web access points such as FTP is more likely to conform to corporate policy.
• 2) It's far easier to integrate a replicated database into proprietary data processing pipelines, with dedicated search strategies and analytical processes. The SureChEMBL web site provides some chemical search capabilities, but cheminformatics is a rich and diverse field, and only a fragment of possible search capabilities are exposed via the general-purpose web site.
• 3) Dedicated resources can be provided for chemical search and analysis, avoiding the need to share public systems with other users. Chemistry searching is a resource-intensive activity, so taking complete control of the hosting is desirable.

Essentially a local replica provides far more flexibility, control, and safety to power users than can be offered by the SureChEMBL web site, hence my client's need.

As SureChEMBL is an open resource, EMBL/EBI stipulated that the result of this project be made available to everyone. As such, the resulting data loading mechanism is available here, under the MIT open source license. This means that anyone can now build their own local replica of the data.

Please note however that the data loading mechanism is provided as-is, with no warranty from myself, or from EBI who have provided access to the underlying data for this project.

The remainder of this article discusses the components and associated data flows, some pros and cons to hosting a replica, as well as potential enhancements that could be made to the data loading mechanism, and to SureChEMBL as a whole.

Components and Data Flows

The following diagram shows the key components, systems, and data flows required to create a local replica of SureChEMBL:

SureChEMBL data client architecture

Components in orange are hosted and maintained by EMBL/EBI. The components in blue must be provisioned and/or developed by the local system administrators or cheminformatics practitioners. The light blue component provides the bridge between the EMBL/EBI systems and the local RDBMS, and can be installed and run according to these instructions.

Patent documents are processed by the SureChEMBL pipeline as they are published, and cause updates to the master database. Then, every night, newly detected chemical annotations are extracted and made available in flat file format via an FTP server. These files are downloaded by the data client, and loaded into the RDBMS using a pre-defined schema.

Once the data has been loaded into the database, it can be queried, searched, extracted, or processed as needed. For example a particular user may decide to add new compounds to a fully-fledged chemistry search engine, or to filter the compounds according to pre-defined criteria and generate alerts if any interesting new compounds are detected.

The database schema has been designed for portability, and tested on both Oracle and MySQL. The following diagram shows the tables and fields in the schema - click the link in the caption to zoom:

SureChEMBL data client schema.

Pros and Cons

If you've read this far, you're probably considering replicating SureChEMBL data in your local environment. If so, there are a few other things to bear in mind.

First, make sure you dedicate enough resources for the RDBMS. The resulting database will be in the order of hundreds of megabytes, and will take days to build rather than hours. The SureChEMBL data client uses an RDBMS because patent-chemistry data is strongly relational. The tradeoff for representing data in this way is typically lower performance and more complex querying and management - this is because the RDBMS manages consistency for you through foreign key relationships, uniqueness relationships, etc.

Second, the resulting dataset is not a one-to-one replica of the data hosted on the SureChEMBL servers, rather it's a "client-facing" version of the data that provides (a) A mapping between chemical compounds and patents, with frequency of occurrence, (b) Chemical representations (SMILES, etc), (c) Chemical attributes such as molecular weight and medicinal chemistry alerts, and (d) Patent metadata such as titles and classifications. You may find that you need to augment the SureChEMBL data from other sources, or request addition of other fields from EMBL/EBI.

Third, bear in mind that the local replica provides a snapshot of the SureChEMBL data, so any data quality improvements made to historic patents won't be reflected in your database; that said, an update or patching mechanism may be provided in the future.

Finally, remember that the SureChEMBL data client can be modified to suit your needs. The client includes, for example, a filtering mechanism that flags certain patents as "life science relevant", based on patent classifications. You may wish to change this based on your own definition of relevance, or to prevent irrelevant patents from being loaded at all.

Where next?

The data loading mechanism is complete as it stands, but there are a few useful extensions that may be added in the future.

In particular, a data patching mechanism would be useful, as it's currently easier to rebuild the replica than to apply changes to the loaded data. It would also be beneficial if there were richer patent metadata, as well as further chemical properties. Finally, it would be helpful if there was a way to seamlessly integrate chemical search such as the JChem cartridge - as it stands there is no default facility in place for searching.

There are also several extensions that could be made to SureChEMBL. These include new types of annotation (such as other biological entities), addition of new patent authorities, as well as data filtering and other quality improvements.

If you have questions regarding the data loading mechanism described in this article, please see my website for contact details. If you have questions related to the wider SureChEMBL system, please contact the SureChEMBL team.