{"id":1057,"date":"2016-12-05T15:48:45","date_gmt":"2016-12-05T15:48:45","guid":{"rendered":"http:\/\/blog.stratio.com\/?p=1057"},"modified":"2023-09-20T13:50:13","modified_gmt":"2023-09-20T13:50:13","slug":"continuous-delivery-depth-2","status":"publish","type":"post","link":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/","title":{"rendered":"Continuous Delivery in depth #2"},"content":{"rendered":"

The not so lean side<\/h2>\n

Remember\u00a0issue #1<\/a>\u00a0published in the summer? We are back with the next part in the series, wearing the hat of Pitfall Harry to look at some of the issues we have come across and how these have impacted our day-to-day job. We also include some tips for overcoming them.<\/p>\n

First things first: Jenkins’ pipelines are an awesome improvement over basic Jenkins funcionalities, allowing us to easily build complex continuous delivery flows, with extreme reusability and maintainability attributes.<\/p>\n

Having said this, pipelines are code. And code is written by human beings. Human beings\u00a0make mistakes. Such errors are reflected as software defects and execution failures.<\/p>\n

This post will take a look at some of the defects, pitfalls and limitations of our (amazing) Jenkins’ pipelines, defining some possible workarounds.<\/p>\n

<\/p>\n

Timeout!<\/h2>\n

The timeout step is the current final defence against hung runs. If a run usually averages X minutes, X*3 is, with almost 100% certainty, a hung run. And as nobody likes zombies crawling around their house, better to dispose of them.<\/p>\n

But – and this is about to change – timeouts won’t stop code enclosed in withEnv or withCredentials steps. There is\u00a0an issue<\/a>, currently being worked on, with\u00a0a github PR<\/a>.<\/p>\n

As 90% of our methods run enclosed in withX elements, we are currently dealing with\u00a0this issue, forcing us to hand-abort runs.<\/p>\n

Workaround:<\/strong>\u00a0Wait and cross fingers. Manual actions applicable when needed.<\/p>\n

NonSerializableException<\/h2>\n

Jenkins serializes objects at several points to be ready for surviving restarts or node failures. This is a really nice feature for overcoming temporal service loss and restarts. The serialization to disk allows unserializing if some actions need to\u00a0be restarted.\u00a0Checkpoints plugin<\/a>\u00a0will probably use this serialization for its own good. Whenever a non-freed non-serializable object exists and Jenkins tries to serialize it, BOOM! NonSerializableException happens.<\/p>\n

A NonSerializableException will be thrown, stopping the job run. Pretty big Impact,\u00a0right?<\/p>\n

Workaround:<\/strong>\u00a0Watch out for iterators, matchers, jsonslurpers, or any other class not implementing the java Serializable interface usage. As it is difficult to infer what class are you working with, as Groovy can be untyped or even inner classes could be unknown (JsonSlurper usesa LazyMap internally not a Serializable), testing will throw a hand.<\/p>\n

@NonCPS<\/a>\u00a0annotation is useful to overcome this, as no serialization will happen on the annotated method. It is not a 100% solution as you cannot invoke pipeline steps inside such methods.<\/p>\n

Testing<\/h2>\n

We need to test to ensure the implemented functionality is properly coded. Corner cases are handled and no NonSerializableException are thrown. But – unfortunately – there is no testing harness for complex pipelines.<\/p>\n

No tests, no fun. To ensure a solid code, testing is a must.<\/p>\n

Workaround:<\/strong>\u00a0Testing\u00a0needs to be done manually, using\u00a0Jenkins’ script console, auxiliary jobs or by\u00a0replaying jobs. Another option is to limit the possible scope of error and to let the run continue.<\/span><\/p>\n

Two left hands<\/h2>\n

Humans write pipeline libraries. As there is no way to locally build the pipeline itself, invalid code could end up in the SCM tool. Badly-written code will lead to errors whilst parsing it, so the affected runs won’t continue.<\/p>\n

Workaround:<\/strong>\u00a0Eat your own dogfood. PRs for each functionality, with its own pipeline.\u00a0Static code analysis<\/a>\u00a0and code reviews will help.<\/p>\n

Node connection loss<\/h2>\n

Remember the NonSerializableException pitfall? It was meant to enable jenkins to survive restart or node reconnections. A problem\u00a0arises when the agent is a cloud-provided one. Such agents will have a variable name (usually with a generated hash) that uniquely identifies each one. This inhibits a job from switching to another agent, even if the first one is not currently alive.<\/span><\/p>\n

Our environment is stacked over a docker swarm, so this happens rather\u00a0frequently.<\/p>\n

Workaround:<\/strong>\u00a0Put up with it\u00a0or propose a change to the docker plugin. Cloudbees have solved this snag at their\u00a0Jenkins Operations Center<\/a>.<\/p>\n

Closure evaluation<\/h2>\n

Pipelines as code are great (have I not already said this?). They allow us to implement complex structures. One of these\u00a0is a Map <String, GString>, and each GString is composed with several lazy evaluated\u00a0GStrings<\/a>\u00a0(who said Inception?), each with another lazy evaluated vars (AKA another GString). This structure, probably coming from\u00a0a CPS issue<\/a>, led us to the wrong top-level GString evaluation. Gstings are objects with two arrays: strings and values. Those arrays get blended on a call or toString invocation, resulting in an eat String object.<\/p>\n

This is probablly the toughest issue we had to overcome and it almost put our project on hold. The data structures we needed were heavily impacted by this issue of\u00a0Lazy evaluation + further GStrings.<\/p>\n

Workaround:<\/strong>\u00a0The first inmediate idea would be to NonCPS the affected code: WRONG! The returned string was also incorrect, cut at the first GString string element (from the strings array).<\/p>\n

We ended up parsing an\u00a0Object with class GStringImpl, and returning\u00a0its arrays of strings and values properly placed. This is known as the famous gStringyHackifier method.<\/p>\n

These are a few of the\u00a0remarkable issues we have had to deal with. We share them with you as a kind of reference in case any of you encounter the same, thus providing tips on how to continue pipelining.<\/p>\n","protected":false},"excerpt":{"rendered":"

Remember\u00a0issue #1\u00a0published in the summer? We are back with the next part in the series, wearing the hat of Pitfall Harry to look at some of the issues we have come across and how these have impacted our day-to-day job. We also include some tips for overcoming them.<\/p>\n","protected":false},"author":2,"featured_media":849,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[686],"tags":[],"ppma_author":[794],"yoast_head":"\nContinuous Delivery in depth (part 2) - Stratio Blog<\/title>\n<meta name=\"description\" content=\"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Continuous Delivery in depth #2\" \/>\n<meta property=\"og:description\" content=\"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Stratio\" \/>\n<meta property=\"article:published_time\" content=\"2016-12-05T15:48:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-20T13:50:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"730\" \/>\n\t<meta property=\"og:image:height\" content=\"312\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@stratiobd\" \/>\n<meta name=\"twitter:site\" content=\"@stratiobd\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\"},\"author\":{\"name\":\"admin\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/af4f5fbbeb95bd7d55f79d9a677e615d\"},\"headline\":\"Continuous Delivery in depth #2\",\"datePublished\":\"2016-12-05T15:48:45+00:00\",\"dateModified\":\"2023-09-20T13:50:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\"},\"wordCount\":873,\"publisher\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif\",\"articleSection\":[\"Product\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\",\"url\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\",\"name\":\"Continuous Delivery in depth (part 2) - Stratio Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif\",\"datePublished\":\"2016-12-05T15:48:45+00:00\",\"dateModified\":\"2023-09-20T13:50:13+00:00\",\"description\":\"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage\",\"url\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif\",\"contentUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif\",\"width\":730,\"height\":312},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.stratio.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Continuous Delivery in depth #2\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#website\",\"url\":\"https:\/\/www.stratio.com\/blog\/\",\"name\":\"Stratio Blog\",\"description\":\"Corporate blog\",\"publisher\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.stratio.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\",\"name\":\"Stratio\",\"url\":\"https:\/\/www.stratio.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png\",\"contentUrl\":\"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png\",\"width\":260,\"height\":55,\"caption\":\"Stratio\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/stratiobd\",\"https:\/\/es.linkedin.com\/company\/stratiobd\",\"https:\/\/www.youtube.com\/c\/StratioBD\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/af4f5fbbeb95bd7d55f79d9a677e615d\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/image\/589aaf4b404b1fe099b09564062c4563\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9b181ae4395243dccaf1c3e3a4749d81?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9b181ae4395243dccaf1c3e3a4749d81?s=96&d=mm&r=g\",\"caption\":\"admin\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Continuous Delivery in depth (part 2) - Stratio Blog","description":"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/","og_locale":"en_US","og_type":"article","og_title":"Continuous Delivery in depth #2","og_description":"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows","og_url":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/","og_site_name":"Stratio","article_published_time":"2016-12-05T15:48:45+00:00","article_modified_time":"2023-09-20T13:50:13+00:00","og_image":[{"width":730,"height":312,"url":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif","type":"image\/gif"}],"author":"admin","twitter_card":"summary_large_image","twitter_creator":"@stratiobd","twitter_site":"@stratiobd","twitter_misc":{"Written by":"admin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#article","isPartOf":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/"},"author":{"name":"admin","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/af4f5fbbeb95bd7d55f79d9a677e615d"},"headline":"Continuous Delivery in depth #2","datePublished":"2016-12-05T15:48:45+00:00","dateModified":"2023-09-20T13:50:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/"},"wordCount":873,"publisher":{"@id":"https:\/\/www.stratio.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif","articleSection":["Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/","url":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/","name":"Continuous Delivery in depth (part 2) - Stratio Blog","isPartOf":{"@id":"https:\/\/www.stratio.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif","datePublished":"2016-12-05T15:48:45+00:00","dateModified":"2023-09-20T13:50:13+00:00","description":"Take a look at some of the defects, pitfalls and limitations of our Jenkins' pipelines, defining some workarounds to build complex continuous delivery flows","breadcrumb":{"@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#primaryimage","url":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif","contentUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2016\/08\/Jenkins.gif","width":730,"height":312},{"@type":"BreadcrumbList","@id":"https:\/\/www.stratio.com\/blog\/continuous-delivery-depth-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.stratio.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Continuous Delivery in depth #2"}]},{"@type":"WebSite","@id":"https:\/\/www.stratio.com\/blog\/#website","url":"https:\/\/www.stratio.com\/blog\/","name":"Stratio Blog","description":"Corporate blog","publisher":{"@id":"https:\/\/www.stratio.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.stratio.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.stratio.com\/blog\/#organization","name":"Stratio","url":"https:\/\/www.stratio.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png","contentUrl":"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png","width":260,"height":55,"caption":"Stratio"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/stratiobd","https:\/\/es.linkedin.com\/company\/stratiobd","https:\/\/www.youtube.com\/c\/StratioBD"]},{"@type":"Person","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/af4f5fbbeb95bd7d55f79d9a677e615d","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/image\/589aaf4b404b1fe099b09564062c4563","url":"https:\/\/secure.gravatar.com\/avatar\/9b181ae4395243dccaf1c3e3a4749d81?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9b181ae4395243dccaf1c3e3a4749d81?s=96&d=mm&r=g","caption":"admin"}}]}},"authors":[{"term_id":794,"user_id":2,"is_guest":0,"slug":"admin","display_name":"admin","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/9b181ae4395243dccaf1c3e3a4749d81?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/1057"}],"collection":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/comments?post=1057"}],"version-history":[{"count":11,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/1057\/revisions"}],"predecessor-version":[{"id":13961,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/1057\/revisions\/13961"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/media\/849"}],"wp:attachment":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/media?parent=1057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/categories?post=1057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/tags?post=1057"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}