Automation, Sytem Engineer

Deploy Go Application Using Jenkins Pipeline

Good Morning, Guys!

Welcome back!

So, I would like to help you to deploy your Golang application to your server using Jenkins Pipeline. But first, what is Jenkins Pipeline? Well, based on Jenkins website, Jenkins pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. So, to use pipeline, We have to install its plugin first from Our Jenkins Plugin configuration page.

Next, If you never using Jenkins pipeline before, You need to know to use Pipeline script which is a Groovy language. To create pipeline script, we can create a file named Jenkinsfile and checked it into project’s source control repository. Or by write it on the Job’s configuration page.

In this example, I’ll create it on the Job’s configuration page. At My pipeline script, I will create 4 stages of deployment which is setup environment, doing test to the code, build the code, and deploy the code to server.

Here it is my pipeline script:

### setup environment: creates directories, sshagent, checking
 ### deployment date, set PATH environment

stage "setup environment"
 node('master'){
 sshagent (credentials: ['2e1b7262-0912-409f-83fb-179860abee79']){
 sh 'ssh -o StrictHostKeyChecking=no -l ec2-user 111.222.123.121'
 sh 'DATE=`ssh ec2-user@111.222.123.121 "date +%Y%m%d%H%M%S"`'
 }
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/bin'){}
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/pkg'){}
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/'){
 sh 'rm -rf *.tar.gz'
 }
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/src/bitbucket.org/[project]/[project_go]'){
 sh 'rm -rf *'
 }
 withEnv(['PATH=$PATH:/opt/go/bin:/usr/local/bin','GOROOT=/opt/go','GOPATH=/var/lib/jenkins/jobs/[your_project]/workspace/']){
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/src/bitbucket.org/[project]/[project_go]') {
 git branch: 'develop', credentialsId: 'c1db80fe-1eab-43f7-bd08-f4f80b706e61', url: 'git@bitbucket.org:rezast/golang.git'
 sh 'go get github.com/spf13/viper'
 sh 'go get gopkg.in/gin-gonic/gin.v1'
 sh 'go get github.com/op/go-logging'
 sh 'go get github.com/dgrijalva/jwt-go'
 sh 'go get github.com/stretchr/testify'
 }
 }
 }

### Run test script

stage "GO Test"
 node('master'){
 withEnv(['PATH=$PATH:/opt/go/bin','GOROOT=/opt/go','GOPATH=/var/lib/jenkins/jobs/[your_project]/workspace/']){
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/src/bitbucket.org/[project]/[project_go]') {
 sh 'go test bill_test.go'
 }
 }
 }
 stage "Build Go!"
 node('master'){
 withEnv(['PATH=$PATH:/opt/go/bin:','GOROOT=/opt/go','GOPATH=/var/lib/jenkins/jobs/[your_project]/workspace/']){
 dir('/var/lib/jenkins/jobs/[your_project]/workspace/src/bitbucket.org/[project]/[project_go]'){
 sh 'go install'
 }
 }
 }

### Send binary to server

stage "Deploy Go"
 node('master'){
 dir('/var/lib/jenkins/jobs/[your_project]/workspace'){
 sh 'tar -czf deploy.tar.gz bin'
 sshagent (credentials: ['2e1b7262-0912-409f-83fb-179860abee79']){
 sh 'scp deploy.tar.gz ec2-user@111.222.123.121:~/'
 sh 'ssh ec2-user@111.222.123.121 "cd ~;tar xzf deploy.tar.gz"'
 echo 'Finished!'
 }
 }
 }

Note to remember: This pipeline script just working until send the binary to the server, to run the binary as a daemon, you should make it on your own, ok? 🙂

Next, lets try to run our deployment!

Result:

At the #5, my deployment is failed, But as you can see at the #6 my deployment is success. You can see the process log by clicking on the stage you want to see. It is more easy to understand rather than the normal Jenkins job, right?

So, that’s all for today 🙂

Hope you enjoy this. Let me know if you have some suggestion! 🙂

Failure is the beginning of success! 😀

See you!

Automation, Sytem Engineer

Integrate Sonarqube and Jenkins

Good day, everyone!

Well, today i’m going to help you integrating your sonarqube and jenkins. If you haven’t installed Sonarqube yet, you may check this link to help you first.

So next, to integrate Sonarqube with Jenkins, we need to install 2 Jenkins plugins they are Sonarqube Plugin and Quality Gates Plugin. Install those plugins via Manage Plugin page on your Jenkins.

After install the plugins, go to Jenkins Configure System page and check the Sonarqube server configurations, you can look at mine down below:

And Quality Gates configuration:

Then, open Jenkins global tool configuration page and look for Jenkins Scanner then add Jenkins scanner:

After all, let’s try to create a Jenkins Job.

On the build section, add Execute Sonarqube Scanner like below:

And on the Post-build Action add Quality Gates like below:

After that, save and try to run your Jenkins Job.

How to prove the Quality Gates?

So, to prove that our quality gates is working, let’s create a new quality gates on our Sonarqube. Here I have 2 quality gates in my Sonarqube. They are Sonarqube Way and Quality Gates Test:

On the first test to run Jenkins Job, we will use the Sonarqube Way quality gates. My project is passed the quality gates as below:

PostBuild-Step: Quality Gates plugin build passed: TRUE
Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
Triggering a new build of new_job_2
Finished: SUCCESS

Next, let’s try to change the quality gates for our project on Sonarqube.

Then, try to run Jenkins job again. We will see a failure notice like below:

PostBuild-Step: Quality Gates plugin build passed: FALSE
Build step 'Quality Gates' marked build as failure
Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
Finished: FAILURE

We can see, on the first run Quality Gates plugin build passed is TRUE and it is triggering a new build for new_job_2, while the second try Quality Gates plugin build passed is FALSE and marked build as failure and it will not triggering any job. PS: You need to add build trigger section on the job configuration setup so if a build is success it will trigger a job to run/build.

So, that’s all for today! 🙂

Please comment for any suggestion/question!

Thank you!!!

Automation, Sytem Engineer

Running Sonarqube on Mac

Good morning, Guys!

Welcome back! 🙂 Finally i have time to write something on my wordpress again 😀

So, this time i’m going to write about Sonarqube and install it on my mac. First of all, what is Sonarqube? Sonarqube is one of open source platform for continuous inspection of code quality. Sonarqube has some features that could make our development will run better, for more information about its features you may check here.

Next, You can install Sonarqube on your Mac, Windows, or linux system. You can check the requirements here. To download Sonarqube you can check here.

After download the archive file and extract it, go to your Sonarqube home directory /bin/[your environment]/

To run Sonarqube service, you can run ./sonar.sh start if everything is ok, it will start the service and run on port 9000. Now you can try to open browser and access it.

Screen Shot 2017-04-18 at 10.00.41 AM.png

So that is just the Sonarqube service that can show your code quality. Now, we need a sonar-scanner to scan our code and some sample project.

Go here to download the scanner and follow the instruction there to install it.

Now, you can try to open to your project directory and add the sonar-project.properties file in it. After that, run sonar-scanner to run the scanning process.

Here my sample project from sonarqube’s website:

javascript-sonar-runner-lcov sudo sonar-scanner
Password:
INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
INFO: Project root configuration file: /Users/rezasetiadi/Documents/setiadi/sonarqube/SonarSource-sonar-examples-07a1bd5/projects/languages/javascript/javascript-sonar-runner-lcov/sonar-project.properties
INFO: SonarQube Scanner 3.0.0.702
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Mac OS X 10.12.3 x86_64
INFO: User cache: /var/root/.sonar/cache
INFO: Load global settings
INFO: Load global settings (done) | time=111ms
INFO: User cache: /var/root/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=16ms
INFO: Download sonar-csharp-plugin-5.7.0.612.jar
INFO: Download sonar-java-plugin-4.5.0.8398.jar
INFO: Download sonar-flex-plugin-2.3.jar
INFO: Download sonar-php-plugin-2.9.2.1744.jar
INFO: Download sonar-scm-svn-plugin-1.4.0.522.jar
INFO: Download sonar-javascript-plugin-2.20.0.4207.jar
INFO: SonarQube server 6.3.1
INFO: Default locale: “en_US”, source code encoding: “UTF-8”
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=16ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=132ms
INFO: Load active rules
INFO: Load active rules (done) | time=972ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=87ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Publish mode
INFO: Project key: org.sonarqube:javascript-lcov-sq-scanner
INFO: ————- Scan JavaScript :: Coverage LCOV :: SonarQube Scanner
INFO: Load server rules
INFO: Load server rules (done) | time=162ms
INFO: Language is forced to js
INFO: Initializer GenericCoverageSensor
INFO: Initializer GenericCoverageSensor (done) | time=0ms
INFO: Base dir: /Users/rezasetiadi/Documents/setiadi/sonarqube/SonarSource-sonar-examples-07a1bd5/projects/languages/javascript/javascript-sonar-runner-lcov
INFO: Working dir: /Users/rezasetiadi/Documents/setiadi/sonarqube/SonarSource-sonar-examples-07a1bd5/projects/languages/javascript/javascript-sonar-runner-lcov/.scannerwork
INFO: Source paths: sources
INFO: Test paths: tests
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: 6 files indexed
INFO: Quality profile for js: Sonar way
INFO: Sensor NoSonar Sensor [php]
INFO: Sensor NoSonar Sensor [php] (done) | time=1ms
INFO: Sensor Coverage Report Import [csharp]
INFO: Sensor Coverage Report Import [csharp] (done) | time=0ms
INFO: Sensor Coverage Report Import [csharp]
INFO: Sensor Coverage Report Import [csharp] (done) | time=0ms
INFO: Sensor Unit Test Results Import [csharp]
INFO: Sensor Unit Test Results Import [csharp] (done) | time=0ms
INFO: Sensor XmlFileSensor [java]
INFO: Sensor XmlFileSensor [java] (done) | time=0ms
INFO: Sensor Analyzer for “php.ini” files [php]
INFO: Sensor Analyzer for “php.ini” files [php] (done) | time=3ms
INFO: Sensor JavaScript Squid Sensor [javascript]
INFO: 4 source files to be analyzed
WARN: Since SonarQube 6.2 property ‘sonar.javascript.lcov.reportPath’ is deprecated. Use ‘sonar.javascript.lcov.reportPaths’ instead.INFO: 4/4 source files have been analyzed

INFO: Unit Test Coverage Sensor is started
INFO: Analysing [/Users/rezasetiadi/Documents/setiadi/sonarqube/SonarSource-sonar-examples-07a1bd5/projects/languages/javascript/javascript-sonar-runner-lcov/report/lcov.dat]
INFO: Integration Test Coverage Sensor is started
INFO: Overall Coverage Sensor is started
INFO: Analysing [/Users/rezasetiadi/Documents/setiadi/sonarqube/SonarSource-sonar-examples-07a1bd5/projects/languages/javascript/javascript-sonar-runner-lcov/report/lcov.dat]
INFO: Sensor JavaScript Squid Sensor [javascript] (done) | time=503ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=15ms
INFO: Sensor Code Colorizer Sensor
INFO: Sensor Code Colorizer Sensor (done) | time=1ms
INFO: Sensor CPD Block Indexer
INFO: org.sonar.scanner.cpd.deprecated.DefaultCpdBlockIndexer@48106381 is used for js
INFO: Sensor CPD Block Indexer (done) | time=1ms
INFO: No SCM system was detected. You can use the ‘sonar.scm.provider’ property to explicitly specify it.
INFO: Calculating CPD for 4 files
INFO: CPD calculation finished
INFO: Analysis report generated in 115ms, dir size=31 KB
INFO: Analysis reports compressed in 19ms, zip size=15 KB
INFO: Analysis report uploaded in 284ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard/index/org.sonarqube:javascript-lcov-sq-scanner
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://localhost:9000/api/ce/task?id=AVt_C73HBpklGzlph9bP
INFO: Task total time: 4.003 s
INFO: ————————————————————————
INFO: EXECUTION SUCCESS
INFO: ————————————————————————
INFO: Total time: 11.533s
INFO: Final Memory: 50M/343M
INFO: ————————————————————————

After execution is success, now let’s check our Sonarqube website:

Screen Shot 2017-04-18 at 10.14.24 AM.png

You can see that our sample project is passed the quality gate provided by Sonarqube. You can see each detail of the scan result. If you find something wrong with the code or your code is not passed the test, go tell your developer to fix it 😀 🙂

So, guys, that’s all for today. Next time i will try to make a post about integrate Sonarqube with Jenkins Server so we could make a better continuous integration in our development.

збогом.

Sytem Engineer

How to Check if a Service is Running and Accessible

Good afternoon, guys 🙂

Today, i’m going to share about how to check if a service is running and accessible from public.  I create this is just to remind us after we installing an application or service that runs on a port, the application is accessible from public.

For the test, for example i installed a nginx for our web server and postgresql for the DB Server. We all know that by default nginx will runs on port 80 and postgresql on port 5432. Then, we want our DB server only accessible from web server.

First thing to know that, we have to change the postgresql configuration file in /var/lib/pgsql/data/pg_hba.conf and /etc/postgresql/8.2/main/postgresql.conf. On pg_hba.conf, we add following line to give access to our network (e.g. 192.168.0.0/24) :

host all all 192.168.0.0/24 trust

While on postgresql.conf file, we change below parameter:

listen_addresses=192.168.0.5

After that all, start the postgresql service.

To check the web server is connected to DB server, we can use telnet command like below from our web server:

telnet [db server hostname] 5432

if connected, it should give us result like below:

Connected to 192.168.0.4
Escape character is '^]'.
Connection closed by foreign host.

After that, we should check it from other host too. It is to make sure no server can access the DB server except the web server.

Beside that, to check port is in use on a server we can use nmap command like below:

nmap 192.168.0.4

If it give us result that port 5432 is open, then we can conclude that our db service is running.

Practice above is always give us a positive return, but what if we got a negative return? Port is already open but we can not access it?

Well, maybe you should check your firewall so you can access both web server or DB server.

If you use Ubuntu, you can simply using below command:

sudo ufw allow 80/tcp

If you use Centos, you can use below command:

sudo iptables -I INPUT -p tcp -m tcp --dport 80 -j ACCEPT

Otherwise, if you use NAT (Network Address Translations) on your environment, make sure you are binding a right port to a right address like below:

Public IP
public port
Private IP
private port
56.234.23.x
81
192.168.0.5
80
56.234.23.x
5433
192.168.0.4
5432

So that’s all, guys. Maybe it sounds so mainstream or basic. But, i hope it would help us to setup our deployment correctly 🙂

Automation, Sytem Engineer

Preparing Jenkins to do Automated Testing

Good afternoon, guys. Back to my routines, it is time to post some knowledge 😀

In this article, i would like to tell you about preparing our Jenkins server to run QA Testing so your QA can automate their tests. In my test, i used my mac as Jenkins server. So if you have another UNIX system, this article might work for you 🙂

There are some steps to make Jenkins can doing QA Automation in Android, iOS, and Web.

In this guide, Jenkins will have one node which is in the same Mac as Jenkins itself.

Jenkins Preparation

  1. Install Jenkins on your mac. You can download jenkins from https://jenkins.io/download/.
  2. After Jenkins is installed, you can configure its plugin. For this article, we need below Plugins:
  • Environment Injector Plugin
  • JDK Parameter Plugin
  • Node and Label Parameter Plugin

Maven 3.3.9

You can download the archive file from https://maven.apache.org/download.cgi, then extract it to /opt/ directory.

After it’s extracted, change the owner of the directory to jenkins and add its path to .zshrc.

sudo chown -R jenkins:wheel apache-maven-3.3.9

Add Path to .zshrc

export PATH=/opt/apache-maven-3.3.9/bin:$PATH

Add Path in Jenkins system configuration:

screen-shot-2017-01-16-at-11-29-35-am

Java

You could install Java using this instruction: https://java.com/en/download/help/mac_install.xml

To verified that java is installed, you can check it by:

java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

You can also add Java’s Path to your environment:

export JAVA_HOME=$(/path/to/java_home)

Android SDK

You can download the latest SDK command line tools from: https://developer.android.com/studio/index.html?hl=sk

Extract the archive file then add it to your $PATH

export ANDROID_HOME=/Users/rezasetiadi/Library/Android/sdk
export PATH=$ANDROID_HOME/platform-tools:$PATH
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/build-tools:$PATH

If Jenkins can’t access Android SDK, you could extract the archive file once again in other directory such as /opt/ and change the owner of the directory then add its Path to Jenkins system configuration.

screen-shot-2017-03-02-at-2-55-29-pm

screen-shot-2017-03-02-at-2-55-35-pm

Appium

You can install Appium on your Mac using npm:

npm install -g appium

To check if your environment is already suit to do Android and iOS automation with Appium, we can install appium-doctor.

npm install -g appium-doctor

After it is installed, check it like below:

~appium-doctor --ios
info AppiumDoctor Appium Doctor v.1.2.5
info AppiumDoctor ### Diagnostic starting ###
info AppiumDoctor ✔ Xcode is installed at: /Applications/Xcode73.app/Contents/Developer
info AppiumDoctor ✔ Xcode Command Line Tools are installed.
info AppiumDoctor ✔ DevToolsSecurity is enabled.
info AppiumDoctor ✔ The Authorization DB is set up properly.
info AppiumDoctor ✔ The Node.js binary was found at: /usr/local/bin/node
info AppiumDoctor ✔ Carthage was found at: /usr/local/bin/carthage
info AppiumDoctor ✔ HOME is set to: /Users/rezasetiadi
info AppiumDoctor ### Diagnostic completed, no fix needed. ###
info AppiumDoctor 
info AppiumDoctor Everything looks good, bye!
info AppiumDoctor
~ appium-doctor --android
info AppiumDoctor Appium Doctor v.1.2.5
info AppiumDoctor ### Diagnostic starting ###
info AppiumDoctor ✔ ANDROID_HOME is set to: /Users/rezasetiadi/Library/Android/sdk
info AppiumDoctor ✔ JAVA_HOME is set to: /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home
info AppiumDoctor ✔ adb exists at: /Users/rezasetiadi/Library/Android/sdk/platform-tools/adb
info AppiumDoctor ✔ android exists at: /Users/rezasetiadi/Library/Android/sdk/tools/android
info AppiumDoctor ✔ emulator exists at: /Users/rezasetiadi/Library/Android/sdk/tools/emulator
info AppiumDoctor ✔ Bin directory of $JAVA_HOME is set
info AppiumDoctor ### Diagnostic completed, no fix needed. ###
info AppiumDoctor 
info AppiumDoctor Everything looks good, bye!
info AppiumDoctor

If there is some item that not checklisted by appium-doctor, you can repair it is installed correctly.

Chrome driver & Safari driver

You can install chrome driver using brew command like below:

brew install chromedriver

To install Safari driver, you can follow this instruction:

https://itisatechiesworld.wordpress.com/2015/04/15/steps-to-get-selenium-webdriver-running-on-safari-browser/

Setting Jenkins Node

Because of Jenkins user in our Mac is headless (has no GUI), we can use our user as Jenkins Node. This configuration will help us to run Web QA Automation.

On the Jenkins configuration create new Jenkins Node as follow:

screen-shot-2017-01-16-at-11-52-32-am

 

screen-shot-2017-01-16-at-11-52-43-am

Make sure this node is connected to the Jenkins server, usually we can just configuring from this setting page.

Make Jenkins Job

Create jenkins job like below:

screen-shot-2017-01-16-at-1-37-53-pm screen-shot-2017-01-16-at-1-38-04-pm

On the build step, you can see the Maven Goals, you can ask the QA team what command to run their script.

Last:

Before run the jenkins job, don’t forget to run the appium server from your terminal and specify which port which you’ll use to run the test.

Note:

If you want to run iOS and Android QA Automation, you need to specify the appium server, make sure you have the same version as the QA.

If you want to run web QA automation, you can select the node you have created before, so that jenkins can open browsers to run automation.

To run iOS QA Automation, you need to install the same XCode as the QA team requirements. If you have multiple XCode on your Mac, you need to set the default XCode using XCode Command Line Tools.

If your iOS test can’t launch the App because of error Instrument Exited with error code 253, you can try to restart your Mac, or if you have multiple XCode, try to reselect the default XCode to use by Appium.

For example, in the first try QA ask to use XCode 7.3. And you have XCode 7.3 and XCode 8.2, try to select the default to 8.2 first, then turn it back to 7.3.