{"id":11678,"date":"2018-09-05T08:24:57","date_gmt":"2018-09-05T08:24:57","guid":{"rendered":"http:\/\/stblog.lunaeme.com\/?p=11678"},"modified":"2023-09-20T15:38:23","modified_gmt":"2023-09-20T15:38:23","slug":"wild-data-part-three-transfer-learning","status":"publish","type":"post","link":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/","title":{"rendered":"Wild Data Part 3: Transfer Learning"},"content":{"rendered":"

Did you know that the word \u201chippopotamus\u201d is a word of Greek origin? Hippos- comes from “horse” and -potamos means “river”. The funny thing here would be to imagine when Greeks run into this animal for the very first time. There was not a word for every single animal around the world, so they probably thought something like \u201cwhat a strange horse…!!! Maybe the river has something to do with it. Got it! It will be a hippo-potamus<\/a>!\u201d<\/p>\n

<\/p>\n

It seems that it is easier combine meanings than to make up a new word. Humans have the inherent ability to transfer knowledge between tasks, as we have already seen in the previous posts<\/a>. Deep Learning was inspired by biological systems (neurons<\/a> and memory<\/a>) and often is inspired in human learning:<\/p>\n

\"Features
Img1. Features of human learning<\/figcaption><\/figure>\n

Transfer Learning<\/h2>\n

Transfer Learning might remind us to some artificial intelligence film scenes like when Neo automatically received his martial arts training during Matrix. However, if we were to compare it is more like the \u201cwax off – wax on\u201d scene in Karate kid. In other words, we recognise and apply relevant knowledge from previous learning experiences. Hence, transfer learning consists in training a base network and reusing some or all of this knowledge (weights) in a related but different task. Thereby, intermediary hidden layers in computer vision are able to learn general knowledge about images(edges, shapes, textures, style\u2026)<\/p>\n

\"Neural
Img2. Neural network and typically small \u201cweights\u201d<\/figcaption><\/figure>\n

In the previous Wild Data series we analyzed Transfer Style<\/a>, where we somehow transferred the style from one image to another through the inner understanding of a pre-trained neural network. We may consider this transfer style like a kind of transfer learning, although we are going to analyze here the typical way of transfer learning.<\/p>\n

Why Transfer Learning?<\/h2>\n

To learn a new language is expensive, but… the more languages we speak, the easier will be to learn another one. In the same way, to train the most complex models in Deep Learning is an intense task. We could start from scratch by initializing the weights with random numbers, but it’s often better use a pre-trained network weights to start with:<\/p>\n

\"Transfer
Img3. Transfer vs No transfer<\/figcaption><\/figure>\n

As you can see, not only will it take less time to get better results, but also less training data will be needed<\/strong> compared to the huge amount of data required to train the full network. In other words, just imagine that you have a smart friend who has studied very hard by understanding the whole syllabus to prepare for an exam. He has prepared very good and organized notes which he lends you and then you pass the exam by using these notes and just studying them the night before. Mmm… I think I take a shine to this guy. In the same way, you could reuse weights you have previously built, or just download<\/a> them from the official website and try to load them into your own model. Also, you could use Keras<\/a>, which makes your life easier by providing you some models for image classification in a very simple way (VGG19<\/a>, ResNet50<\/a>, InceptionV3<\/a>…) Note: be aware of the input “image_size” that are given to each pre-trained model.<\/p>\n

So, should I always use Transfer Learning?<\/h2>\n

Yes, you should use it whenever it makes sense. No kidding! Transfer learning looks pretty awesome but you have to take into account the context in which it will be used. If, for example, you need to carry out an infrared light problem, it will work horribly if we use a network that has been trained in visible spectrum. Thus, the strategy will mainly depend on the amount of data available and the similarity of this data:<\/strong><\/p>\n

\"Transfer
Img4. Transfer Learning Strategy<\/figcaption><\/figure>\n

It is clear that you should have a lot more data for training the pretrained model than your new task. Besides, low level features from the pretrained model could be helpful for training the new one, and then it is necessary to remove the last Fully-Connected layer of the network, and to add instead the classifier adapted to your new task. On the other hand, fine-tuning is a destructive process that forgets previous knowledge<\/strong> in order to train a new one quickly or waste less time during training. You might as well freeze the weights for all or lower layers of the network and just fine-tune the layers which are not frozen.<\/p>\n

\"Fine-tuning
Img5. Fine-tuning some layers while freeze others<\/figcaption><\/figure>\n

In conclusion, most of companies could benefit from shipping faster to production<\/a> of Deep Learning models already trained. Computer Vision has been the field which has seen better improvements, but recent research (ELMo<\/a>, ULMFiT<\/a>, and the OpenAI transformer<\/a>) have demonstrated that transfer learning can be used too on a wide range of Natural Language Processing tasks. It should be pointed out that other particular application of transfer learning could be simulations, because of gathering data and training a model in the real world could be expensive or just too dangerous (for instance, some situations like autonomous driving<\/a>).<\/p>\n

If you want to learn more about Deep Learning take a look at our last Data Science Meetup<\/a> we organized at Campus Madrid:<\/p>\n

– Video<\/a>
\n– Presentation<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.<\/p>\n","protected":false},"author":1,"featured_media":13547,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[686],"tags":[19],"ppma_author":[795],"yoast_head":"\nWild Data - Transfer Learning - Deep Learning<\/title>\n<meta name=\"description\" content=\"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Wild Data Part 3: Transfer Learning\" \/>\n<meta property=\"og:description\" content=\"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"Stratio\" \/>\n<meta property=\"article:published_time\" content=\"2018-09-05T08:24:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-20T15:38:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1300\" \/>\n\t<meta property=\"og:image:height\" content=\"820\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Stratio\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@stratiobd\" \/>\n<meta name=\"twitter:site\" content=\"@stratiobd\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Stratio\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\"},\"author\":{\"name\":\"Stratio\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/d0377b199cd052b17e15c9ba44c45ab7\"},\"headline\":\"Wild Data Part 3: Transfer Learning\",\"datePublished\":\"2018-09-05T08:24:57+00:00\",\"dateModified\":\"2023-09-20T15:38:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\"},\"wordCount\":909,\"publisher\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg\",\"keywords\":[\"Big Data\"],\"articleSection\":[\"Product\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\",\"url\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\",\"name\":\"Wild Data - Transfer Learning - Deep Learning\",\"isPartOf\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg\",\"datePublished\":\"2018-09-05T08:24:57+00:00\",\"dateModified\":\"2023-09-20T15:38:23+00:00\",\"description\":\"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage\",\"url\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg\",\"contentUrl\":\"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg\",\"width\":1300,\"height\":820},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.stratio.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Wild Data Part 3: Transfer Learning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#website\",\"url\":\"https:\/\/www.stratio.com\/blog\/\",\"name\":\"Stratio Blog\",\"description\":\"Corporate blog\",\"publisher\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.stratio.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#organization\",\"name\":\"Stratio\",\"url\":\"https:\/\/www.stratio.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png\",\"contentUrl\":\"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png\",\"width\":260,\"height\":55,\"caption\":\"Stratio\"},\"image\":{\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/stratiobd\",\"https:\/\/es.linkedin.com\/company\/stratiobd\",\"https:\/\/www.youtube.com\/c\/StratioBD\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/d0377b199cd052b17e15c9ba44c45ab7\",\"name\":\"Stratio\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/image\/bb38888f58c2bb664646155f78ae6ccc\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e3387ad00609f34a56d6796400eb8191?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e3387ad00609f34a56d6796400eb8191?s=96&d=mm&r=g\",\"caption\":\"Stratio\"},\"description\":\"Stratio guides businesses on their journey through complete #DigitalTransformation with #BigData and #AI. Stratio works worldwide for large companies and multinationals in the sectors of banking, insurance, healthcare, telco, retail, energy and media.\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Wild Data - Transfer Learning - Deep Learning","description":"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/","og_locale":"en_US","og_type":"article","og_title":"Wild Data Part 3: Transfer Learning","og_description":"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.","og_url":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/","og_site_name":"Stratio","article_published_time":"2018-09-05T08:24:57+00:00","article_modified_time":"2023-09-20T15:38:23+00:00","og_image":[{"width":1300,"height":820,"url":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg","type":"image\/jpeg"}],"author":"Stratio","twitter_card":"summary_large_image","twitter_creator":"@stratiobd","twitter_site":"@stratiobd","twitter_misc":{"Written by":"Stratio","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#article","isPartOf":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/"},"author":{"name":"Stratio","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/d0377b199cd052b17e15c9ba44c45ab7"},"headline":"Wild Data Part 3: Transfer Learning","datePublished":"2018-09-05T08:24:57+00:00","dateModified":"2023-09-20T15:38:23+00:00","mainEntityOfPage":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/"},"wordCount":909,"publisher":{"@id":"https:\/\/www.stratio.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg","keywords":["Big Data"],"articleSection":["Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/","url":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/","name":"Wild Data - Transfer Learning - Deep Learning","isPartOf":{"@id":"https:\/\/www.stratio.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg","datePublished":"2018-09-05T08:24:57+00:00","dateModified":"2023-09-20T15:38:23+00:00","description":"Transfer learning consists in training a base network and reusing some or all of this knowledge in a related but different task.","breadcrumb":{"@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#primaryimage","url":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg","contentUrl":"https:\/\/www.stratio.com\/blog\/wp-content\/uploads\/2018\/09\/Wild-Data-Part-3-Transfer-Learning.jpg","width":1300,"height":820},{"@type":"BreadcrumbList","@id":"https:\/\/www.stratio.com\/blog\/wild-data-part-three-transfer-learning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.stratio.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Wild Data Part 3: Transfer Learning"}]},{"@type":"WebSite","@id":"https:\/\/www.stratio.com\/blog\/#website","url":"https:\/\/www.stratio.com\/blog\/","name":"Stratio Blog","description":"Corporate blog","publisher":{"@id":"https:\/\/www.stratio.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.stratio.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.stratio.com\/blog\/#organization","name":"Stratio","url":"https:\/\/www.stratio.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png","contentUrl":"https:\/\/stratio.com\/blog\/wp-content\/uploads\/2020\/06\/stratio-web-logo-1.png","width":260,"height":55,"caption":"Stratio"},"image":{"@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/stratiobd","https:\/\/es.linkedin.com\/company\/stratiobd","https:\/\/www.youtube.com\/c\/StratioBD"]},{"@type":"Person","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/d0377b199cd052b17e15c9ba44c45ab7","name":"Stratio","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stratio.com\/blog\/#\/schema\/person\/image\/bb38888f58c2bb664646155f78ae6ccc","url":"https:\/\/secure.gravatar.com\/avatar\/e3387ad00609f34a56d6796400eb8191?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e3387ad00609f34a56d6796400eb8191?s=96&d=mm&r=g","caption":"Stratio"},"description":"Stratio guides businesses on their journey through complete #DigitalTransformation with #BigData and #AI. Stratio works worldwide for large companies and multinationals in the sectors of banking, insurance, healthcare, telco, retail, energy and media."}]}},"authors":[{"term_id":795,"user_id":1,"is_guest":0,"slug":"stratioadmin","display_name":"Stratio","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/e3387ad00609f34a56d6796400eb8191?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/11678"}],"collection":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/comments?post=11678"}],"version-history":[{"count":16,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/11678\/revisions"}],"predecessor-version":[{"id":15720,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/posts\/11678\/revisions\/15720"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/media\/13547"}],"wp:attachment":[{"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/media?parent=11678"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/categories?post=11678"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/tags?post=11678"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.stratio.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=11678"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}