Retrieve the Source Code (FTP, Subversion and GitHub)

Introduction

The first step in working with the C++ Toolkit is getting the source code, which can be retrieved from GitHub or from a Subversion repository. This chapter describes both methods and the use of utility scripts that can help getting only the necessary source code components.

If you are interested in downloading source code from the C Toolkit instead of the C++ Toolkit, please see Access to the C Toolkit source tree Using CVS.

Chapter Outline

The following is an outline of the topics presented in this chapter:

• Public Access to the Source Code via GitHub

• Read-Write Access to the Source Code via Subversion (NCBI only)

  • NCBI Source Tree Contents

  • Source Code Retrieval under Unix

    • Retrieval of the C++ Toolkit Source Code Tree

      • Checking Out the Development NCBI C++ Toolkit Source Tree

      • Checking Out the Production NCBI C++ Toolkit Source Tree

      • svn_toolkit_tree: Quickly checking out the whole Toolkit source tree

      • svn_core: Retrieving core components

      • import_project: Retrieve Source for an Existing Project

      • update_core: Update the Portable and Core Components

      • update_projects: Check out and Update Sources of Selected Projects

  • Source Code Retrieval under MS Windows

    • Retrieval via Script in Windows

  • Source Code Retrieval under Mac OS X

• Public Access to the Source Code via FTP

• Source Tree Structure Summary

Public Access to the Source Code via GitHub

The current TRUNK and public releases of the C++ Toolkit is available on GitHub at https://github.com/ncbi/ncbi-cxx-toolkit-public.

At this time it is a read-only mirror of our master SVN repository, updated nightly. Note however that one can submit Pull Requests to it (assumed to be under the overall C++ Toolkit license only!). We will review and consider such PRs for merging into our SVN repository – from which they will make its way back into this GitHub mirror.

Read-Write Access to the Source Code via Subversion (NCBI only)

Note: This section discusses read-write access to the Subversion repository, which is only available to users inside NCBI. For public access, see the section on read-only access.

Subversion client installation and usage instructions are available on separate pages for UNIX, MS Windows, and Mac OS systems.

For a detailed description of the Subversion Version Control System please download the book “Version Control with Subversion” or run the command svn help on your workstation for quick reference.

The following is an outline of the topics presented in this section. Select the instructions appropriate for your development environment.

• NCBI Source Tree Contents

• Source Code Retrieval under Unix

  • Retrieval of the C++ Toolkit Source Tree

    • Checking Out the Development NCBI C++ Toolkit Source Tree

    • Checking Out the Production NCBI C++ Toolkit Source Tree

    • svn_core: Retrieving core components

    • import_project: Retrieve Source for an Existing Project

    • update_core: Update the Portable and Core Components

    • update_projects: Check out and Update Sources of Selected Projects

• Source Code Retrieval under MS Windows

  • Retrieval via Script in Windows

• Source Code Retrieval under Mac OS X

NCBI Source Tree Contents

The NCBI C++ Toolkit Subversion repository contains all source code, scripts, utilities, tools, tests and documentation required to build the Toolkit on the major classes of operating systems (Unix, MS Windows and Mac OS).

Source Code Retrieval under Unix

Retrieval of the C++ Toolkit Source Code Tree

This section discusses the methods of checking out the entire source tree or just the necessary portions. An important point to note is that the entire NCBI C++ tree is very big because it contains a lot of internal projects. There are also numerous platform-specific files, and even platform-specific sub-trees, which you will never need unless you work on those platforms. Therefore it is frequently sufficient, and in fact, usually advisable, to retrieve only the files of interest using shell scripts. The relevant scripts are located in /am/ncbiapdata/bin, but the best way to get them into your PATH is to make sure you have developer in the facilities line of your ~/.ncbi_hints file.

The following sections discuss the checkout process in more detail:

• Checking Out the Development NCBI C++ Toolkit Source Tree

• Checking Out the Production NCBI C++ Toolkit Source Tree

• svn_toolkit_tree: Quickly checking out the whole Toolkit source tree

• svn_core: Retrieving core components

• import_project: Retrieve Source for an Existing Project

• update_core: Update the Portable and Core Components

• update_projects: Check out and update Source of Selected Projects

Note: To facilitate the creation of a new project, use the script new_project which generates new directories and makefiles for the new project from templates, but does not involve checking out files.

Checking Out the Development NCBI C++ Toolkit Source Tree

You can check out the entire development NCBI C++ source tree from the repository to your local directory (e.g., foo/c++/) just by running:

cd foo
svn checkout https://svn.ncbi.nlm.nih.gov/repos/toolkit/trunk/c++

For internal projects use:

cd foo
svn checkout https://svn.ncbi.nlm.nih.gov/repos/toolkit/trunk/internal/c++

Caution: Be aware that sources checked out through the development source tree have the latest sources and are different from the public release that is done at periodic intervals. These sources are relatively unstable “development” sources, so they are not guaranteed to work properly or even compile. Use these sources at your own risk (and/or to apply patches to stable releases).The sources are usually better by the end of day and especially by the end of the week (like Sunday evening).

Checking Out the Production NCBI C++ Toolkit Source Tree

Besides the development NCBI C++ source tree, there is the C++ Toolkit “production” source tree that has been added to the public Subversion repository. This tree contains stable snapshots of the “development” C++ Toolkit tree. Please note that these sources are lagging behind, sometimes months behind the current snapshot of the sources.

You can check out the entire “production” NCBI C++ source tree from the public repository to your local directory by running:

svn co https://svn.ncbi.nlm.nih.gov/repos/toolkit/production/latest/c++

This repository path corresponds to the latest production build of the Toolkit. If you want to check out sources for an older production build, please specify the exact date of that build, for example:

svn co https://svn.ncbi.nlm.nih.gov/repos/toolkit/production/20031212/c++

where 20031212 is the date when this specific build was originated. You can easily find out the available production builds by running

svn ls https://svn.ncbi.nlm.nih.gov/repos/toolkit/production

This command will print directories under production/, which correspond to the production builds.

svn_toolkit_tree: Quickly checking out the whole Toolkit source tree

Usage:

svn_toolkit_tree <archive> <new_dir>

Checking out the whole Toolkit source tree using a Subversion client can take 15 minutes or more. However, the script svn_toolkit_tree produces the same result in under a minute. The svn_toolkit_tree script combines a daily archive with an update of the working copy to bring it up-to-date. This produces the same set of files and revisions as running svn checkout, but in much less time. Besides speed, the differences between using a Subversion client and the svn_toolkit_tree script include:

• The svn_toolkit_tree script may not be compatible with your Subversion client. If your client is older than the version used to create the archive, you may not be able to access the archive.

• The svn_toolkit_tree script will not accept the name of an existing directory.

The archives that were available at the time of this writing (October 2014) were:

Run the script with no arguments to find the most up-to-date list of supported archives.

For example, to retrieve the current TRUNK version of the “core” part of the C++ Toolkit tree (the part without the GUI and INTERNAL projects), run:

$ svn_toolkit_tree core cpp
/net/snowman/vol/projects/ncbisoft/toolkit_trees/trunk-core.tar.gz -> cpp
Updating cpp...

$ ls cpp
compilers  configure  doc  include  scripts  src

svn_core: Retrieving core components

Usage:

svn_core <dir> <branch> [--export] ... [--<platform>]

The NCBI C++ Toolkit has many features and extensions beyond the core of portable functionality. However, one often wants to obtain a set of core sources that is free of non-portable elements, and the svn_core script performs this task across the range of supported platforms. Options to the basic command allow the developer to further tailor the retrieved sources by including (or excluding) certain portions of the Toolkit.

For usage help, run svn_core without arguments.

Note: svn_core is not available on Windows.

Table 1 describes the arguments of svn_core. Only the target directory and SVN branch arguments are mandatory.

Table 1. svn_core Arguments

Some directories are always checked out, regardless of command-line arguments. These are shown in Table 2. (All paths are relative to the repository path https://svn.ncbi.nlm.nih.gov/repos/toolkit/trunk/c++.)

Table 2. List of the directories that are always checked out

Other directories may or may not be checked out, depending on the <branch> and <platform> options. These are shown in Table 3.

Table 3. Directories that may be checked out depending on branch and platform options

Still other directories may be checked out depending on the --with/--without-<feature> options. These are shown in Table 4.

Table 4. Directories that may be checked out depending on –with/–without options

import_project: Retrieve Source for an Existing Project

Usage:

import_project [options] <SVN_relative_tree_path> [builddir]

The import_project script imports the source from a single project, and configures the resulting tree.

In many cases, you work on your own project which is a part of the NCBI C++ tree, and you do not want to check out, update and rebuild the entire NCBI C++ tree. Instead, you just want to use headers and libraries of the pre-built NCBI C++ Toolkit to build your project.

The shell script import_project will check out your project’s src and include directories from the repository and create temporary makefiles based on the project’s customized makefiles. The new makefiles will also contain a reference to the pre-built NCBI C++ Toolkit.

For example:

import_project serial/datatool

will check out the datatool project from the NCBI C++ tree (trunk/c++/{src,include}/serial/datatool/), and create a makefile Makefile.datatool_app that uses the project’s customized makefile Makefile.datatool.app. Now you can just go to the created working directory c++/src/serial/datatool/ and build the application datatool using:

make -f Makefile.datatool_app

update_core: Update the Portable and Core Components

Usage:

update_core [--no-projects] [<dirs>]

Once you have obtained the core C++ Toolkit sources, with svn_core or otherwise, the local copies will become out of sync with the master SVN repository contents when other developers commit their changes. update_core will update your local core source tree with any changed files without the side-effect of simultaneously checking out non-core portions of the tree. Subdirectory */internal does not get updated by this script.

The --no-projects switch excludes any Windows or MacOS project files from the update. Specifically, those subdirectory names of the form *_prj are skipped during the update when this flag is set.

The list [<dirs>], when present, identifies the set of directories relative to the current directory to update. The default list of updated directories is:

• .

• compilers

• doc

• include

• scripts

• src

Note that the default list is not pushed onto a user-supplied list of directories.

update_projects: Check out and update Source of Selected Projects

Usage:

update_projects <project-list> [<directory>]

Script update_projects facilitates the original retrieval and subsequent updates of selected parts of the Toolkit tree. Because the source code and makefiles are distributed over more than one subdirectory under repository path trunk/c++, this script assembles the set of required files and places them in your local C++ source tree.

The projects to be retrieved (or updated) must be specified in the command line as the <project-list> parameter. Its value can be either of the following:

• Explicit specification of the pathname of the project listing file. This project listing file can contain project directory names as well as references to other project listings and must be formatted according to the simple syntax used by the configure script.

• Specify one of the standard project names. Standard projects are those whose project listing files are located in one of the system directories, which are trunk/c++/scripts/projects and trunk/c++/scripts/internal/projects. When a project name is specified on the command line, the “.lst” extension is added to it and the resulting file name is searched for in the above mentioned system directories.

The parameter to update_projects indicates the target directory where the sources will be checked out to and where the project will be configured and built. This parameter is optional and is set to the current directory by default.

Source Code Retrieval under MS Windows

1. In NCBI, the SVN clients must be set up and ready to use. Ask Systems if you don’t have the client installed on your workstation. If you are working outside of NCBI, then you can download a subversion client from a third-party provider. Apache does not endorse any third-party binary packages, but does maintain a list. Run the Subversion installer and follow the instructions. The latest version may not come with an executable installer though. In this case, please unpack the zip archive with the latest Subversion binaries to a local directory, for example C:\Program Files\svn-win32-1.4.2. Change the PATH environment variable so that it points to the bin subdirectory under your Subversion installation directory, for example set PATH=%PATH%;C:\Program Files\svn-win32-1.4.2\bin

2. Start your favorite command shell. Change current directory to the designated working directory. At the command prompt, type:svn co https://svn.ncbi.nlm.nih.gov/repos/toolkit/trunk/c++

3. Modify source files as required. Refer to Svnbook for the documentation on particular Subversion commands. Monitor your changes using svn diff, synchronize your working copy with the trunk using svn update, and finally commit them using svn commit.

The rest should be the same as when using Subversion under UNIX systems. See Source Code Retrieval under Unix.

Retrieval via Script in Windows

Some of the source code retrieval scripts are available in Windows:

• svn_toolkit_tree

• import_project

• update_projects

Source Code Retrieval under Mac OS X

Download and install the latest Subversion binaries for MacOSX from https://subversion.apache.org/.

The rest should be the same as when using Subversion under UNIX systems. See Source Code Retrieval under Unix.

Public Access to the Source Code via FTP

Old releases of the Toolkit can be downloaded via FTP. At present they are probably of historical interest only.

• FTP Download Now

• Available FTP Archives: Select the archive for your system. When the dialog box appears, choose the destination in your file system for the downloaded archive. Note: With some browsers, you may need to right-click-and-hold with your mouse and use the 'Save Link As...', 'Copy to Folder...', or similar options from the drop-down menu to properly save the archive. For a current list of the source code archives for different operating system/compiler combinations consult the current Release Notes.

• Unpack the Source Archive

  • Unix and Macintosh Systems
    The Unix distributions have been archived using the standard tar command and compressed using gzip. When unpacked, all files will be under the directory ncbi_cxx--<version_number>, which will be created in the current directory. (Caution: If ncbi_cxx--<version_number> already exists, tar extraction will overwrite existing files.) To unpack the archive: gunzip -c ncbi_cxx--*.tar.gz | tar xvf -

  • Windows Systems
    The Microsoft Windows versions of the source distribution have been prepared as self-extracting executables. By default a sub-folder ncbi_cxx--<version_number > will be created in the current folder to contain the extracted source. If ncbi_cxx--<version_number > already exists in the folder where the executable is launched, user confirmation is required before files are overwritten. To actually perform the extraction, do one of the following:

    • Run the executable from a command shell. This will create the sub-folder in the shell’s current directory, even if the executable is located somewhere else.

    • Double-click on the archive’s icon to create ncbi_cxx--<version_number > in the current folder.

    • Right-click on the archive’s icon, and select 'Extract to...' to unpack the archive to a user-specified location in the filesystem.

Source Tree Structure Summary

To summarize the Getting Started page, the source tree is organized as follows:

• The top-level has configuration files and the directories include/, src/, scripts/, compilers/ and doc/

• The src and include directories contain “projects” as subdirectories. Projects may contain sub-projects in a hierarchical fashion.

• src/ additionally contains makefile and meta-makefile templates.

• Projects contain “modules” and various customized makefiles and meta-makefiles to control their compilation.