Blog

Tutorial: Migrating Data from MySQL to MongoDB

by | Jan 20, 2015 | Developer | 2 comments

In the next tutorial you will learn how to migrate data from MySQL to MongoDB. We will show you how to do it using Spark step by step. From creating a configuration for the player RDD to the installation guide for prerequisites components.
Easy and intuitive!

PREREQUISITES*:

  • MongoDB (versión 2.6 recommended)
  • MySQL 5
  • Java 7+
  • Maven 3+
  • Spark 1.2
  • Deep-Spark


TUTORIAL TO LOAD A DATASET TO MySQL:

Create schema:

Create tables:

Populate tables:

Running Spark-shel:

 

USING SPARK STEP BY STEP:

Necessary imports:

Creating a configuration for the player RDD and initialize it:

Creating the RDDs that represent the data set in MySQL:

Map teams to pair with (team id, team):

Map players to pair with (team id, player) and group by team_id

Creating a configuration for the mongodb result RDD and initialize it:

Transforming the joined result to the desirable structure in mongodb:

Save the RDD in MongoDB


USING SPARK WITH OUR EXAMPLE PROJECT:

https://github.com/robertomorandeira/deep-example 
Just make a git clone a run our java or scala example

Java example:
https://github.com/robertomorandeira/deep-example/blob/master/src/main/java/FootballMigrationApp.java 
Scala example:
https://github.com/robertomorandeira/deep-example/blob/master/src/main/scala/FootballMigrationAppScala.scala 


CHECKING DATA.

Connect to mongodb, normally

You can see the data loaded in mongoDB


INSTALLATION GUIDE FOR PREREQUISITES COMPONENTS:

MongoDB

MySQL Server

Java 7

Maven

X