upgrading from jenkins 1 to jenkins 2

The first Jenkins LTS (long term support) release fro Jenkins 2.0 came out this week.  It is Jenkins 2.7.1 and is available for download here. CodeRanch is running 1.640 and we decided to upgrade shortly after the LTS release came out. There have been numerous weekly releases for 2.0. We wanted the LTS release so it is a little more stable.

Backup

Another moderator at CodeRanch installed Jenkins and Sonar originally. Unfortunately, they are mixed together in the data directory. Anyway, I started by taking a backup in case I want to go back to 1.X at any point or mess it up/ I’m not terribly worried because our Jenkins install has 5 jobs on it. Worse case, I can do a fresh install and re-create them. I want to try upgrading first though since it is easier (at least in theory.)

tar -cvf jenkins1-20060709-bkp.tar *

The tar file was 1.2 GB. This is because it includes the job workspaces and Sonar.

Install

Conveniently our Jenkins install is just the war. So I backed up the old war and uploaded the new one:

mv jenkins.war jenkins1-bkp

Starting up with Jenkins 2

Loading Jenkins showed a nice banner with a big button to click

jenkins2-splash

So I clicked the “upgrade now” button. That led to a prompt to ask if I wanted to install some pipeline related plugins.

I said yes so they installed.

Then I got another button that Jenkins was ready. Still looks the same and my jobs are still there.

jenkins-ready

When I went to look at the plugins that were installed, I saw “Warning: This Jenkins instance requires a restart. Changing the state of plugins at this time is strongly discouraged. Restart Jenkins before proceeding.”. Which makes sense. Odd the upgrade process didn’t prompt me to restart. I did it myself at this point.

Existing Jobs

The five existing jobs ran the same way as they did before upgrade.  That odd phrasing was because what really happened was:

  • Three jobs were successful
  • One job hasn’t been run in ages and was clearly an old experiment. I deleted it.
  • One job has been failing for a few days and nobody noticed. It doesn’t have email notification and is triggered by the main job. (It was failing because it was relying on an Ant 1.9 feature and Jenkins was using Ant 1.8.) I upgraded Ant and then it was good. I also added email notification so that doesn’t happen again.

Jenkins 2 claims to be fully backwards compatible. As near as I can tell, this is true. I didn’t run into any issues related to migration. I even felt comfortable deleting the giant backup file at this point.

What did change

What I noticed very quickly

  1. The job configuration screen now tab navigation which takes you right to a section on the page.
  2. The configuration of external tools is now under Maven Jenkins > Global Tool Configuration instead of Manage System.

Actually using pipelines

Granted Pipelines were available in Jenkins 1. Looking at the jobs we had, we have three distinct steps (the main build runs CI and Sonar and then a JavaDoc build runs.) It would be good to have the Sonar one as a top level citizen and equal to the JavaDoc one. I decided to try to re-write the whole thing in Groovy (rather than calling the existing jobs) to see how hard it would be. I also split the Sonar step out into its own stage. This would have been more difficult to do as a separate job because I’d have had to pass the Sonar revision number and hack at it to use the same workspace. (I know how to do both those things – after all the original JavaDoc job uses an absolute path to the Ant script to run in the original workspace – but still.)

As an aside, I had trouble finding the workspace.

Figuring out email notification was a pain. In particular, getting it to email on every failure and first success just like the default used to be. But the rest worked pretty easily and end to end it only took me a couple hours (and 37 test builds) – it’s a pretty simple set of jobs. And now the email list is one place for all the JForum stages so we won’t have the JavaDoc build failing and nobody knowing again. The script wound up being:

def emailRecipients = 'xxx@gmail.com'

node {
 // hack to get workspace variable - https://issues.jenkins-ci.org/browse/JENKINS-33511
 env.WORKSPACE = pwd()
 
 try {
 stage 'Build'
 // poll SVN and check out updates if changes
 checkout poll: true, scm: [$class: 'SubversionSCM', additionalCredentials: [], excludedCommitMessages: '', excludedRegions: '', excludedRevprop: '', excludedUsers: '', filterChangelog: false, ignoreDirPropChanges: false, includedRegions: '', locations: [[credentialsId: 'e10831de-302f-4c1d-9518-fa7fb81bab9e', depthOption: 'infinity', ignoreExternalsOption: true, local: '.', remote: 'https://xxxx/svn/xxx']], workspaceUpdater: [$class: 'UpdateUpdater']]
 // call Ant
 def antHome = tool 'apache-ant-1.9.7'
 sh "${antHome}/bin/ant clean compile jacoco dist-work"
 step([$class: 'JUnitResultArchiver', testResults: 'qa/reports/*.xml'])
 
 stage 'Sonar'
 sh "${antHome}/bin/ant sonar -DprojectVersion=${env.SVN_REVISION}"
 
 stage 'JavaDoc'
 sh "${antHome}/bin/ant javadocs"
 // archive only if run successfully
 step([$class: 'JavadocArchiver', javadocDir: "${env.WORKSPACE}/docs/api", keepAll: false])
 
 // without this, you don't get emails
 currentBuild.result = 'SUCCESS' 
 
 } catch (err) { 
 // without this, you don't get emails
 currentBuild.result = 'FAILURE' 
 throw err
 } finally {
 echo "currentBuild.result: ${currentBuild.result}"
 // send email whether job failed or not
 step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailRecipients, sendToIndividuals: false])
 }
}

 

sonarqube and the scm plugin

I tried enabling the SCM integration from our CodeRanch Sonar install to our SVN repository. It didn’t quite work out the way I was hoping. But I learned stuff so it makes for a good blog entry.

Wait? The SCM Activity Plugin is deprecated?

If you search for Sonar SCM plugin, you get to the plugin documentation page. Which appears to be useful. Except that it says the plugin has been deprecated since Sonar 5.0. I stopped reading at that point and puzzled over it. If you continue to read, it says “This plugin is deprecated since SonarQube 5.0 which has built-in support for SCM information and which relies on independent plugins to cover SCM providers.” Ah. So it is deprecated because it is now a core feature. That explains why the functionality shows up on our Sonar.

Time to enable

I generated another SVN user and set the username/password in Sonar. I then set “disable the SCM sensor” to false and ran our Jenkins build. And I got this. Well, it went on for pages, but you get the idea:

[sonar:sonar] Sensor SCM Sensor
[sonar:sonar] SCM provider for this project is: svn
[sonar:sonar] 1404 files to be analyzed
[sonar:sonar] 0/1404 files analyzed
[sonar:sonar] Missing blame information for the following files:
[sonar:sonar] * /data/vhost1/ciengine.javaranch.com/data/jobs/JForum/workspace/src/main/java/net/jforum/util/bbcode/BBCodeHandler.java
[sonar:sonar] * /data/vhost1/ciengine.javaranch.com/data/jobs/JForum/workspace/src/main/java/net/jforum/util/concurrent/Executor.java

Ok. So the credentials are good as Sonar can connect to see how many files there are.  Google and SVN plugin page both suggested the problem was not being on Subversion 1.5 or lower. I looked around and that didn’t appear to be the case.  svnadmin —version returned 1.7.8 and the repository was on 1.6. So I thought maybe the mismatch was the problem and tried to run svnadmin upgrade svn (svn is the name of the folder containing our repository on disk). Then this happened and I had to recover from it.

Then it worked

After I upgraded the Subversion repository (or whatever the hell I did by accident), I was able to commit and trigger another Jenkins/Sonar build. I saw the person who committed lines of code in each file. And what’s really cool is that it WENT BACK IN TIME. I see the committers that aren’t active on CodeRanch anymore and haven’t contributed code in years. That is really cool.

What I was hoping it would do vs what it actually does

I wanted to see the “scm blame” output when reviewing files in Sonar. That works and I see it as being useful. I can see if something came from me (or another current developer.) Or if it came from someone no longer involved in the code base.

I wanted to see coverage on new code. This works depending on how you define new code. If you define it as “touched” code, it works well. You also have to remember to use a differential view so Sonar knows how far back to consider “new” code. This makes sense. I typically use the last 7 days as my view so “new” means “touched in the last 7 days”. Unfortunately, we don’t currently run our integration tests in Jenkins so this isn’t useful for the back end layer. I had started dealing with that. Maybe I should finish :).

I had read about the “Developer Cockpit” and seeing a view of your own contributions. I hadn’t realized that was a paid enterprise feature so not helpful to us at CodeRanch.  Without the cockpit, you can still go to “My account” to see a table with “leaks” (issues you created in the past week) and a link to all of your issues. Unfortunately Sonar thinks half the issues are mine and the others aren’t assigned:

  1. I was the one who committed the initial JForum fork we made which means every single open source issue is treated as mine.  (I don’t remember if I did or not, but I led that project so I wouldn’t be surprised.)
  2. Most of the developers don’t have accounts on Sonar. Only the admins do. Everyone else has been using it anonymously. (It’s behind an Apache password wall so not public.)

On the bright side, you *can* filter issues by author now. Which is the last committer. But still helpful. If the code is new, the last committer is someone who might look at it. If the committer is someone who isn’t around anymore, that’s informative.

 

 

 

 

upgrading a svn repository and recovering

While trying to integrate Sonar with Subversion, I had a suspicion that it wasn’t working because of either the version of the repository or a failed upgrade in the past. So on a whim*, I ran the upgrade command:

-bash-4.1$ svnadmin upgrade svn
Repository lock acquired.
Please wait; upgrading the repository may take some time…

Upgrade completed.

svn is the directory with our repository on CodeRanch. This happened almost instantaneously so I figured it was on the latest and nothing happened. Well, something happened. The permissions changed on a few files in the svn directory. I got this message:

Some of selected resources were not committed.
svn: E204900: Commit failed (details follow):
svn: E204900: Can’t open file ‘/home/vhost1/svn.javaranch.com/svn/db/txn-current-lock’: Permission denied
svn: E175002: MKACTIVITY of ‘/svn/!svn/act/bc61d1cc-5501-0010-ba7e-45a47496be2e’: 500 Internal Server Error (https://svn.javaranch.com)

Uh oh. I was able to commit before I started. Long story short, I fixed it by iterating through the errors and wound up just having to run:

chmod g+w txn-current*
 chmod g+w txn-protorevs

I tested by pulling, committing, tagging and deleting a tag. Everything seems to work.

Disclaimer

* If you work at a company, please don’t upgrade things on a whim nor allow your staff to do. I would never do that at my real job. CodeRanch operates more like the wild, wild west. Which is appropriate given the name and the fact that everything is volunteer driven and the culture is one that allows for some risk. I was able to fix this in about 15 minutes so no harm done. But if you are being paid to do something or can’t tolerate the risk, be careful. Do things on a development server. Plan. Tell people. Then these unpleasant moments where you realize you’ve made it so nobody can commit don’t happen.