Feature Update: Mr Scaler Now Supports Scaling of Core Nodes!

We are very excited to announce some feature announcements for our Mr Scaler product. Mr Scaler is an integration service with AWS EMR (Elastic Map Reduce). You can use Mr Scaler to launch Core and Task nodes with up to 80% savings per node. Mr Scaler can also automatically scale your nodes for you based on existing CloudWatch EMR metrics. You can either “wrap” an existing EMR cluster or clone them as new ones. If you are currently using Elastic Map Reduce and are not familiar with our cost saving integrations please check out our original announcement.
If you use Mr Scaler today you are probably already scaling your Task nodes with spot instances today with this service. Today, we are officially announcing our support for scaling of core nodes with Mr Scaler. You can now use this feature to have considerable savings for both task and core nodes. With this feature you can choose between using on-demand instances or spot instances.

Let’s create a new Mr Scaler cluster:

1. From the EMR console in AWS, choose an existing EMR cluster that you would like to clone. This can be a currently running EMR cluster or one that was terminated. Copy down the ID of this EMR cluster
2. From the Spotinst console go to “Mr.Scaler”
3. Click on “Launch Mr Scaler”
4. Select “Clone” and paste in the id of the EMR cluster from step one above. Now, select the subnets that you would like to use and enter the instance type, lifecycle, target/min/max for each of the instance groups *Master is defaulted to 1. Click “Next” at the bottom right of the page.
5. Configure the scaling policies for your Task and Core nodes. You can add scaling policies by clicking on “Add Policy”. Click “Next” at the bottom right of the page to proceed.
6. On the review page review the configuration. Sample below.

   {
     "mrScaler": {
       "name": "Test",
       "region": "us-west-2",
       "strategy": {
         "cloning": {
           "originClusterId": "j-327CXLAY7ML9Q"
         }
       },
       "compute": {
         "availabilityZones": [
           {
             "name": "us-west-2a",
             "subnetId": "subnet-2b68a15c"
           },
           {
             "name": "us-west-2b",
             "subnetId": "subnet-e01bb085"
           }
         ],
         "instanceGroups": {
           "masterGroup": {
             "target": 1,
             "instanceTypes": [
               "c3.xlarge"
             ],
             "lifeCycle": "ON_DEMAND"
           },
           "coreGroup": {
             "instanceTypes": [
               "i2.xlarge",
               "i2.2xlarge"
             ],
             "lifeCycle": "ON_DEMAND",
             "capacity": {
               "target": 1,
               "minimum": 1,
               "maximum": 5
             }
           },
           "taskGroup": {
             "instanceTypes": [
               "c3.xlarge",
               "c4.xlarge",
               "c4.2xlarge",
               "c3.2xlarge"
             ],
             "capacity": {
               "target": 1,
               "minimum": 0,
               "maximum": 5
             },
             "lifeCycle": "SPOT"
           }
         },
         "tags": null
       },
       "scaling": {
         "up": [
           {
             "policyName": "up Scaling Policy 1",
             "metricName": "AppsPending",
             "statistic": "average",
             "unit": "count",
             "threshold": 1000,
             "action": {
               "type": "adjustment",
               "adjustment": 1
             },
             "namespace": "AWS/ElasticMapReduce",
             "dimensions": [
               {
                 "name": "JobFlowId"
               }
             ],
             "period": "300",
             "evaluationPeriods": 1,
             "cooldown": 3000,
             "operator": "gte"
           }
         ]
       },
       "coreScaling": {}
     }
   }

7. After reviewing the configuration click on “Create”. The creation process will take a few minutes.
8. You can now see the Mr Scaler dashboard. Notice that all nodes have been launched and should be healthy now.
   
9. You can also modify the capacity of your nodes from the “Actions” menu at the top right of the dashboard.
   

We hope you enjoy the new Mr Scaler features. If you are using EMR today give it try, you can save up to 80% vs on-demand prices when you launch your core or task nodes with Spot instances.

-The Spotinst Team