{"id":306,"date":"2015-04-28T08:47:20","date_gmt":"2015-04-28T08:47:20","guid":{"rendered":"http:\/\/blog.stratio.com\/?p=306"},"modified":"2023-09-20T13:43:21","modified_gmt":"2023-09-20T13:43:21","slug":"a-spark-based-analytics-solution-for-online-advertisers","status":"publish","type":"post","link":"https:\/\/www.stratio.com\/blog\/a-spark-based-analytics-solution-for-online-advertisers\/","title":{"rendered":"A Spark-based analytics solution for Online Advertisers"},"content":{"rendered":"
\n
\n

This post contains the winning solution for the Stratio challenge 2015 developed by Marco Piva, Leonardo Biagioli, Fabio Fantoni and Andrea De Marco (BitBang).<\/p>\n

The work describes the data model and the architecture of a Big Data Analytics solution that can help online advertisers to get fast answers for their analytical questions about impressions, clicks and purchases,basing on the Stratio Challenge requirements.<\/p>\n

The emphasis is placed on the technologies that have been used, with particular focus on Spark and Spray.<\/p>\n<\/div>\n<\/div>\n

<\/p>\n

\n

Big Data analytics tools in Online Advertisement\u00a0<\/h2>\n

We’re living in a world of 24\/7 connectivity, accessing our content on our own terms, and we like it that way. Around the globe, 76% of respondents in a Nielsen online survey say they enjoy the freedom of being connected anywhere, anytime. While consumers love this flexibility, it represents a huge challenge for brands and online advertisers vying for our attention in a fragmented viewing arena.<\/p>\n

Online Advertising is a relatively new discipline, born in the early 1990s along with the rise of the World Wide Web. After a few years dominated by basic web banners displayed on the major web sites, online companies started to experiment and to consolidate multiple formats and strategies to advertise their products, also defining standards and best practices about display advertising, search engine marketing, etc. According to IAB Internet Advertising Revenue Report, Internet advertising revenues in the USA reached the high of $12.4 billion during the third quarter of 2014, with a year-over-year growth that is continuing since 2010.<\/p>\n

In this highly-demanding and ultra-competitive industry tools that support analysis and intelligence are essential: nobody wants to spend money on poor publishers, and every advertiser needs to have a scientific way to demonstrate how effective campaigns are in terms of ROAS (Return On Advertising Spending).<\/p>\n

Analytics tools are becoming more and more complex and sophisticated in order to provide valuable insights on the marketing activities and results.<\/p>\n

There are three major requirements for analytics tools today:<\/p>\n

\n
    \n
  1. Real-time: queries must run on datasets that are updated in real-time;<\/li>\n
  2. Volume: collected data are frequently quantified in terms of terabytes (TBs) if not petabytes (PBs) on a daily basis;<\/li>\n
  3. Velocity: queries on very large datasets must return results in a short time, basing on business needs.<\/li>\n<\/ol>\n<\/div>\n<\/div>\n
    \n
    Stratio Challenge requirements:<\/div>\n
    The solution must meet the following requirements:<\/div>\n
      \n
    1. Provide the number of impressions, clicks and purchases in a given timeframe;<\/li>\n
    2. Provide the metrics at point 1. grouped by country, user-agent, campaign, advertiser or publisher;<\/li>\n
    3. Provide combination of two or more of the dimensions above at point 2.;<\/li>\n
    4. Receive queries and new data at the same time, without any impacts on performance;<\/li>\n
    5. Have one hour as time granularity;<\/li>\n<\/ol>\n<\/div>\n
      \n
      Data Model:<\/div>\n
      Data are modelled using an Entity-Relationship diagram, along with an additional basic schema that includes the assumptions provided by the Stratio Challenge.<\/div>\n
      \"challenge_datamodel\"<\/a><\/div>\n
      \u00a0<\/div>\n
      \u00a0<\/div>\n
      \n

      The Entity-Relationship diagram is designed using the composition relation (instead of aggregation) to emphasize the strong link between these three entities.<\/p>\n

      Due to this choice, a Click could not exist without its “parent impression” in the same way a Purchase could not exist without its “parent Click”.<\/p>\n

      Regarding to the cardinality, a single Impression can lead to multiple Clicks (0..N) but every Click must be generated by a single Impression (1..1); Click and Purchase are ruled by the same cardinality behavior.<\/p>\n

      The following schema shows the assumptions that exist between Impression, Click and Purchase.<\/p>\n<\/div>\n

      \"challenge_figure2\"<\/a><\/div>\n
      \u00a0<\/div>\n
      \u00a0<\/div>\n<\/div>\n
      \n
      Architecture:<\/div>\n
      The main technologies used to build the architecture are the following:<\/div>\n