Sourcefabric Manuals

 English |  Español |  Français |  Italiano |  Português |  Русский |  Shqip

TagesWoche Newscoop 4.4 Implementation

development platform

To have a better understanding of how changes to the code move through the development work-flow, get committed and finally deployed to the system running in production, here you can learn what is what in the platform we use to develop customer projects.

Platform Components

In a general way, when working on a project you will find your assignments in the issue tracker (JIRA), you will have working instances of the projects in a personal development environment (personal container), you will pull/push the code from/to Git repositories (Stash), and you will see whether your code builds the project successfully or not by checking the continuous integration server (Bamboo).

JIRA

JIRA is the tracker system we use in Sourcefabric to create and organize issues, plan and build our products and implement customer projects. In our platform, you can find JIRA at http://dev.sourcefabric.org/

Logging your work

JIRA is also the place where you should log your work: the amount of hours you spend working on your assignments. This is particularly important for project implementations.

Personal Containers

A personal container is a Linux Container (a.k.a. LXC) system, set up to provide an individual development environment for developers when working on project implementations.

The usage of a personal container removes the hassle for the developer of setting up local instances every time she/he is joining a project. The developer though has always the option of running the development environment locally in her/his personal computer but setting up a project instance to be able to work on it is her/his entire responsibility, and this is often a very time-consuming task. On the other hand, using a personal container allows the developer to focus on the deliverable right away as a working development environment for the project is pushed to her/his container.

  • Every developer in Sourcefabric is given a personal container on-request.
  • The personal container is hosted in a remote system managed by Sourcefabric's Systems team.
  • The personal container is accessible via SSH and the developer has sudo access for a complete control of her/his environment.
  • Project instances in a personal container are accessible via a URL that looks like: https://<developer-name>.sourcefabric.net (e.g. https://holman.sourcefabric.net)

Multiple project instances can be deployed to a single personal container. Putting it into different words, if you are a developer and you participate of different projects, the platform will deploy for you working instances of all those to your personal container and you will be able to start working on the code right away. Even better, instances of the same project will be created for you automatically whenever you create a new branch to work on a new feature, improvement or bug-fix.

Stash

Stash is the Git Repository Management system used in Sourcefabric to control the source code for customer projects that usually require private repositories. You can find Stash at http://stash.sourcefabric.org/

Stash is like having a GitHub or Bitbucket system, installed on our own servers. Why? Simple, Stash allows us to have as many private repositories as we need and at the same time provides full integration with JIRA, Confluence and Bamboo. If you know how to work with GitHub or Bitbucket, using Stash will be just as easy.

Most importantly, Stash provides the platform that support our development work-flow for customer implementations. No code gets merged into the Git repositories without going through Stash. Collaborative work, communication, code reviews, all that and more is possible thanks to Stash.

  • Newscoop projects developed using a single repository to hold the code for the Website's theme are located under the Newscoop Implementation project in Stash http://stash.sourcefabric.org/projects/NCIMP
  • Customer projects with more than one repository have a corresponding Stash project and all Git repositories required are grouped together under it.
  • Every developer in Sourcefabric must make sure to setup an account in Stash. Check the Developer's Guide to learn how to do it.
  • We advise developers to use Stash to only host stuff related to customer projects. This is not a policy though and the developer has total freedom on her/his own Stash account.

Solutions which are of general interest and aim to be released under an open-source license (e.g. themes, plugins, modules, etc) should not be developed in Stash, you should rather use the public organization for the product in GitHub https://github.com/<product>/.

Bamboo

Bamboo is the tool used as Continuous Integration (CI) and Continuous Delivery (CD) platform for customer project implementations. You can find Bamboo in our platform at http://bamboo.sourcefabric.org/

Bamboo can be seen as a notification system on project builds. Once a project is setup in Bamboo, Bamboo tries to build it every time a change to the code is submitted, as a result Bamboo informs whether the build was successful or not. It is a great tool to support developers and QA on automatically and quickly finding issues in the code.

Not necessarily every project has a plan setup in Bamboo for Continuous Integration. The Team Leader will communicate the development team whether Bamboo CI is used in the project or not.

Development Environments

In Sourcefabric, software for customer projects is developed in a four tier model: Development, Integration, Pre-Live and Live. Let's see what each of these environments represents.

Development

(a.k.a. personal environment)

The code from Stash repositories together with the instances of projects deployed to a personal container constitute the Development environment. The place where the code gets first written or modified by the developer.

A personal development environment is entirely owned by the developer, in consequence all what happens here is her/his responsibility. The developer can try here radical changes to the code as those will not affect the rest of the development team.

It is in this environment also where the work done by the developer gets reviewed for the first time.

For in-progress code:

  • The developer invite QA testers and other developers to contribute and participate of her/his environment, for the sake of testing and code reviewing.
  • In the same way, project managers, product owners or any other stakeholder are invited to review the partial implementation and provide feedback to the developer.

For fully finished work:

  • Once the developer considers the work is done, she/he must invite some other members of the team for reviewing: QA, Developers, Project Manager and Product Owner/Customer.
  • QA will perform the corresponding tests that suit best the task in hand.
  • Developers will code review the changes.
  • The Project Manager and Product Owner/Customer will perform a usability check.
  • For new features, an acceptance must be given by each reviewer: QA, Developer(s) and Project Manager and/or Product Owner/Customer. Only then the original developer will submit her/his work to Integration.
  • For bug fixes, the acceptance is given by QA, and in cases where the issue affects user experience the Project Manager and/or Product Owner/Customer may also be requested for giving the Ok. Only then the original developer will submit her/his work to Integration.

To submit code changes to the Integration environment the developer must open a Pull Request to the corresponding repository/branch in Stash.

Inviting more people to your development environment to review partial work is not mandatory, it is entirely up to you. However, inviting others is advised. Getting feedback early in the process of development only saves you time and allows you to improve the quality of the code and the implementation before finally pushing it to the next environment.

Integration

The Integration environment is a common place where all developers commit their code changes. The major goal here is to combine and validate the work of the entire development team so that it can be tested before promoting it to the Pre-Live environment.

Developers are the owners of this environment. QA has no incidence in this environment at least it is required as it may happen in some exceptional cases.

  • Issues found at this stage are fixed in the Development environment and re-submitted to Integration.
  • Once developers are happy with the current state and behaviour of the code, that new version is then promoted and deployed to the Pre-Live environment.
  • The Lead Developer deploys the new version to the Pre-Live environment by merging the code to the corresponding repository/branch via a Pull Request in Stash.

Pre-Live

(a.k.a. pre-production or staging environment)

Pre-Live is an environment that mimics the Production environment. It is the last check-in point before deploying the new version of the code to Production.

This environment is owned by QA and it is where the final iteration of testing happens. The QA team thoroughly test the code and validates all changes made and, in general, the integrity of the system. Any found issue is reported back to the development team.

Once the QA team accepts and declares the code as "Ok to release", the Lead Developer runs the deployment to Production by merging the code into the corresponding Stash repository/branch.

Live

(a.k.a. production)

This is the environment facing the end-users, nothing happens here, the system must work trouble-free, that's it.

  • Developers only push changes to the Development and Integration environments. Changes are committed directly by the developer to her/his personal development environment and via Pull Requests to the Integration environment.
  • Changes must not be pushed directly to Pre-Live or Production environments, never ever. This is to preserve the integrity of the source code repository and so the versions running on all environments. The way the changes to the code make it to these environments is by merging from Development.
  • Any developer can merge her/his work to the Integration environment. This happens though only after successful reviews and receiving the corresponding acceptance.
  • Only the Lead Developer can deploy new versions to the Pre-Live and Live environments. This is a task of the Release Manager, however, in Sourcefabric there is not such role, therefore the Release Manager is the Lead Developer. There can be different Lead Developers for different projects, or even different Lead Developers for the same project at different stages. Regardless the conditions, it is always the Lead Developer in charge the only one deploying new versions to these environments.

Git Repositories Structure

The structure of repositories is project-dependant. We try to make this as generic as possible so that getting the code is a no-brainer and you don't need to spend time asking around what to do every time you join a project. Here you will find some guidelines that should work most of the times.

The repositories here refer to the ones used on customer-specific development, for example: a Newscoop theme, a Newscoop plugin, a Booktype module, etc. and it is assumed the project is built on top of an official released version of a product (e.g. Airtime, Booktype, Newscoop, Superdesk.)

Newscoop-based Projects

Every Newscoop-based customer implementation project has a corresponding Stash project where Git repositories can be found. In a typical Newscoop project you can usually find the following type of repositories:

Themes: Repositories which hold all the files that conform a Newscoop theme including template files, CSS files, images, Javascript scripts, etc. One repository holds one and only one theme. These repositories are named using the convention:

theme-<ThemeName>

where theme- is a prefix used to clearly indicate the repository holds code for a theme, and <ThemeName> is the name given to the theme, always using capital cases as in the examples below.

e.g. theme-ZentralAnchorage, theme-YemenLightBlue

Plugins: Repositories which hold all the files that conform a Newscoop plugin. One repository holds one and only one plugin. These repositories are named using the following convention:

plugin-<PluginName>

where plugin- is a prefix used to clearly indicate the repository holds code for a plugin, and <PluginName> is the name given to the plugin, always using capital cases as in the examples below.

e.g. plugin-WorldWeather, plugin-FacebookLogin


Depending on the requirements of the project there may exist other types of repositories. If that is the case, the Developer Leader and Project Manager will accordingly inform the team about it.

Branching

In all cases, every repository will have the following basic branching model:

master
env/prelive
env/live

master is the main development branch, while env/live and env/prelive represent environment branches:

  • master is the default working branch and at the same time holds the in-development source code used in the Integration Environment.
  • env/prelive holds the source code that represents the version being used in the Pre-Live Environment.
  • env/live holds the source code that represents the latest stable version being used in the Production Environment.

Other branches may exist in the repository, in those cases the conventions for the branching model presented here apply.

You, as a developer, should always base your work on the version of the code contained in the master branch.

All Together

You are John Doe and you are working on a project named foobar. You were assigned to develop two new features ft-1234 and ft-4321. This is how your personal development environment should look like:

  • Your personal container has been setup by the Systems team and is available as johndoe.sourcefabric.net
  • You have gone to your Stash account and you have forked the theme-Awesome repository from the foobar project.
  • You do not work directly on the master branch of your repository (a good practice), rather you create two feature branches ft-1234 and ft-4321, and you push those branches to your fork in Stash.
  • You have three instances --automatically created- now in your Development environment, which you can browse via:
    http://master.foobar.johndoe.sourceefabric.net,
    http://ft-1234.foobar.johndoe.sourcefabric.net
    and
    http://ft-4321.foobar.johndoe.sourcefabric.net
  • You keep committing code to your branches and pushing it to Stash and you can verify your changes in the corresponding instances.
  • Whenever you feel happy with the state of your code and you get approval from reviewers, you open a Pull Request to submit your work to the Integration environment.

Remember to invite others (developers, QA stuff, stakeholders) to check out your feature instances so that they can provide valuable feedback early in the process.

There has been error in communication with Booktype server. Not sure right now where is the problem.

You should refresh this page.