• Version control system and web integration projects
• Front-End Task Automation – Part 1
• Front-End Task Automation – Part 2
• Putting Continuous Integration (CI) into Practice 1/2
• Putting Continuous Integration (CI) into Practice 2/2
• Architecture and operating of web applications in the cloud environment (Part 1)
• Architecture and operating of web applications in a cloud environment (Part 2)
• Continuous Integration – Cure for Human Error in Deployment
• Stay Up to Date – Continuous Integration Notification System

Although the choice of specific tools supporting the implementation of the principle of continuous integration into the organization engaged in development is not associated with continuous integration as such in any way, in another interpretation we will stick to terminology of Git version system, Git flow version workflow and principles of semantic versioning.

Concept of continuous integration

In an organization where there is development of projects or products of larger scale in the team, developers typically work on particular parts of the assignment (“features” or “hotfixes”), which are necessary to be systematically combined into one whole of the project while maintaining the integrity (functionality) of already existing work. Continuous Integration is based on continuous performance of acts of integration of source code, testing, building and deployment in response to each change to the source code of the project submitted by the developer and for the use of tools for support of the development and testing by compliance with the established procedure automatedly.

System of control of versions of source code

The cornerstone of deploying continuous integration in an organization is the consistent use of version system for ALL the code of projects, and ideally incl. runtime configuration (with an exception of configurations specific for particular target runtime environment) and database configuration as defined by data structures – in case of relational databases of DDL* commands, for other types of repositories then other descriptive files, usually in XML or JSON format).

(* Data Definition Language – a language for describing data structures (typically database tables)

It is also appropriate to version not only program and configuration code of the projects themselves in the repository, but also meta files, which help to uniformly manage the process of development and integration, e.g. single configuration of Integrated development environment (IDE) of the team, files controlling building of the project (see below) and files configuring this process (project build configuration).

Version systems are generally divided into centralized (CVCS) based on server-client architecture and distributed (DVCS) not requiring server service because they generally suffice with any network-accessible file repository. Among centralized VCS are included CVS, SVN (Subversion) or SourceSafe, among distributed are included e.g. Git or Mercurial, where especially Git has observed a vast expansion not only in the world of open source in recent years.

Central source code repository

Central source code repository represents a network location or better service that solves:

• hosting – physical location of repositories of code of used version system,
• centralization of developed code for transmission and control,
• security settings (restricting access to specific repositories for specific users or better to user groups)
• code control,
• connection of tools for cooperation in development of the project.

Utilizing cloud service eliminates overheads associated with maintaining own solutions to a central repository. In contrast, when using own hardware resources the code remains within the organization’s infrastructure and can thus be considered to be stored more safely (let us ignore the fact that statistically the information leakage occurs the most just inside the corporate network infrastructure). The technical aspect for selecting the type of operation of the service may be also the speed of connection, where the availability of the service located in the internal network of the company is higher in orders of magnitude than for external services.

Centralization brings advantages both for developers and manager roles involved in the project. Developers have one destination for takeover of current code of the project and the subsequent submission of work and thus there is a space for them to focus on their core specialization that is production of functional quality code. On the other hand, staff focused managerially have clearly designated place where they can check the progress of the course of development of the project at any time.

After submitting the project code, its control typically follows. Submitting of work may then take the form of a request for acceptance of set of changes (commits) in the project code (so called pull request), when the developer cannot directly affect the version of the project in a remote repository, but a proposed change by him/her must pass the quality control (code review). The actual control of the code has both the dimension of a control by a responsible person and also automated tests are used to check the submitted code. For more details see below in the chapter Automated testing and code control.

It is advantageous if not even necessary to solve the above mentioned parts of the development by means of specialized existing services / tools and in the sense of hosted (on-demand) service or application which you install on your own infrastructure and operate so called on-premise. Hosted services are represented by Github and Bitbucket, where the organization has a web-based user interface for:

• comfortable management of safety,
• control of pull requests,
• code review,
• history display
• possibility of integration with other services (for tracking of requests, automated deployment etc.)

A tool for solution on own hardware resources may be Stash for Linux (on-premise similar to Bitbucket cloud service) GitWeb, Gitolite or Bonobo Git Server or GitStack for Windows.

Packaging and dependency management (PDM) tool

Within the development infrastructure in addition to determining a central repository of source codes a need arises to storage and organization of packages of already compiled code, either within the organization or independently outside of it or possibly within packaged source code that does not change – it concerns scripting application environments and / or distribution of code for the needs of code assist feature in the development environment (IDE) and the need to clearly define the dependence of the code of the project on these packages. It is therefore necessary to use a tool that allows you to configurationally define the necessary dependencies of the source code (i.e. different frameworks and libraries needed to run the application) and put the resulting compiled application into the form of a distributable package. While in some application platforms the use of this tool is inherent (e.g. Apache Maven not only for Java-based projects, Ruby Gems for Ruby or NPM Node.js/Javascript based projects) in the world of web development on PHP platform its existence in the form of a Composer tool is still a relative novelty and a number of open source projects still does not consider it, nevertheless its use is increasingly promoted.

Each PDM tool commonly brings the following features into the development

• Defines the configuration file of the project for definition of dependencies including repositories, in which it is possible to obtain the dependent packages, and other conditions for the formation of application, usually in XML or JSON format
• Defines format/form of the package of reusable source or compiled code (according to application platform) incl. its meta definition (analogy for project configuration)
• Usually has one or more own public repositories in which packages with popular open source code are made available (in case of Composer tool it is Packagist service)
• A tool for command line (CLI) for controlling the life cycle of the project following the chosen archetype, typically thus
  • Creation of directory structure of the project following the chosen application framework
  • download of all dependencies necessary for development (source codes for code assistance in IDE) or running of the project
  • update of downloaded dependencies at change of configuration
  • build (package) with the possibility of loading into runtime environment (application) or repository of packages (libraries); within building then performing a series of operations that depend on each other

Continuous Integration tool – CI

Continuous Integration tool ensures all the automatable tasks within implementation of CI process i.e. using or in cooperation with above mentioned tools for source code management and controlling of dependencies, its building and package. There are many tools available as open-source as well as commercial, let us mention the best known

• Jenkins CI (successor to the formerly popular Hudson CI) – http://jenkins-ci.org
• JetBrains TeamCity – https://www.jetbrains.com/teamcity/
• Atlassian Bamboo – https://www.atlassian.com/software/bamboo
• Team Foundation Server (TFS) – http://msdn.microsoft.com/en-us/vstudio/ff637362.aspx

Some are directed to use in connection with a specific application platform, others are platform-neutral. CI tool can be operated in both traditional on-premise model or use them as a cloud service. On-premise deployment may be a necessity in some cases, typically when the source code repositories and / or target runtime environments for deploying of the application are placed in a private network of the organization and they cannot be made accessible to the cloud service due to security policies.

The basic property of CI tool is the definition of parametrized tasks, which can disintegrate into individual partial steps. The task may be subsequently done either by running through a web interface of CI tool or through its API by calling upon an event in another system (typically). Tasks may also be combined to a higher logical units. Thus CI tools, to a considerable extent, reiterate the principles of separate building command line tools of Ant, Maven or Phing type with which they may also complement (a step in the building task may call a building tool controlled by building configuration file (build config file), which may be a part of version code of the project or is stored on the CI server independently from it).

Automation of building

Building means an activity which by downloading of the source code and configuration files incl. definition of dependencies and a code required by this definition for translation and / or running of the project will create a form of a project executable in runtime environment and usually in the form of a package, but it may be only a workspace directory created on CI server for further use. Depending on the chosen application technology a translation may be a part of building, in case of PHP it is usually installation or update of built form of the project by Composer tool. Automated building is usually done using CI tool based on a change (a new commit) in one of the main branches in the repository of the source code of the project. A successful automated building of the project then may be an input (trigger) of subsequent automated deployment. In case of a product (application or library) the CI process usually ends with automated building (finished package is delivered to CI server for further use).

Automation of deployment

Based on successful automated deployment and testing of project the CI tool may immediately or independently in a separate task, proceed to automated deployment into target environment. Set of environments which are available in the organization for deployment of the project should be clearly defined and the set of environments which are available in the organization for deployment of the project need to be unanimously understood in the context of development. These environments usually stem from the life cycle of the project.

• Internal test for deployment of current (HEAD) form of the project in the development. In case that shared server environment is used for development of web projects (which is, in case of PHP development, widespread practice in bigger organizations), then the internal test is usually implemented in the same server environment, in which developers develop, by which equivalent running conditions are ensured for it, as a local instance of the project of developers. Development team (incl. testers and internal product owner, delivery manager or project manager continually supervising the process of development) work with the internal test.
• Preview, sometimes also UAT or simply testing environment. There is usually a deployment into the preview from release or hotfix branch of the source code of the project, possibly a separate preview branch may be defined to which developers merge changes, which they want to build into preview environment by means of automation of project deployment. Primarily the Client (product owner) works with preview environment and he/she checks (accepts) incorporated changes in them.
• Production or live environment.

Automated deployment reduces if not directly does not need inputs by developers into runtime environments within the process of deployment, which positively contributes to stable behaviour of the applications.

Automated testing and code control

According to the time moment of launch, tests can be divided into groups pre-commit (before submission), pre-deploy (before deployment) and post-deploy (after deployment into the target environment).

Pre-commit tests

Control of syntactic validity and coding conventions (formatting, documentation of the source code)

Pre-deploy tests

Pre-deploy tests relate to the source code and can be implemented before building or within it (thus typically after all the necessary dependencies are prepared by a package tool). These test may be launched in response to integration of changes in the code in system for administration of the source code into some of the main branches (develop, master) of the main repository of the project. In response to the nature of the test or strategy of integration they can be launched even before the change will be integrated into the branch (so called pre-commit) or after it (post-commit).

• Launching of unit tests implemented within the project for testing individual features (typically using PHPUnit framework)

Post-deploy tests

Post-deploy tests may be implemented only in response to a successful automated deployment as they require an application running in the target environment. These are typically:

• Automated user tests created in Selenium IDE tool and launched by CI tool in Selenium server environment
• Integration tests verifying functionality of the system

Automation of testing makes it obvious and ever present part of the development process. Compared to traditional testing implemented ad-hoc (or neglected) the CI process may require the presence of tests while forcing the team to respond to their failures (be it by a repair of the source code or by an update of the test in response to required implemented functional changes).

Main contributions of deployment of continual integration into the process of making web solutions

Automation of individual repetitive tasks of building to deployment of the application eliminates occurrence of errors associated with manually performing them. In addition, an automated process is always described at least in the form of configuration files or configuration settings of the tools that perform it, when these configurations are managed by limited group of persons (role CI/deployment manager only) and in addition it is usually possible to apply version control system to them as well and thus have a control in the whole team over when and who changed such a meta definition of the process.

Automated implementation of tests ensures continually running testing of the project in response to ongoing change at the level of the source code (typically thus integration of commit into some of the main branches of the project) or even independently on it (post-deploy tests may continuously verify the integrity of running application in the chosen target runtime environment)

Continually running process of building and deployment ensures that in response to the work submitted by the developers and workflow of the project mapping the individual development phases to target runtime environments there are always current tested versions of the project in these environments. At the same time automated deployment facilitates extraordinary remedies, e.g. allows to easily deploy the previous version in case that despite implemented automatized or possibly manual tests the project shows problems in the given environment.

A concept of continual integration is naturally conveniently complemented by agile, interactive methodologies of the software development (Scrum).

Continual integration tool gathers and according to the needs / rules of the organization further distributes information on the result of automated task via available information channels (e-mail, instant messaging, possibly SMS messages). Thus it contributes to the awareness of each member of the development team as well as persons of interest standing outside the team (development manager, project management, product owner) about the course of the process of controlled change in the web project.