Configure Shard/Sharding mongodb

Steps to configure Sharding in MongoDB on Linux


1. Configure 9 different servers

mongod --dbpath /home/replica11 --port 27011 --logpath /home/replica11/rep11.log --logappend --fork --bind_ip_all --replSet replicaset1 --shardsvr

mongod --dbpath /home/replica12 --port 27012 --logpath /home/replica12/rep12.log --logappend --fork --bind_ip_all --replSet replicaset1 --shardsvr

mongod --dbpath /home/replica13 --port 27013 --logpath /home/replica13/rep13.log --logappend --fork --bind_ip_all --replSet replicaset1 --shardsvr

mongod --dbpath /home/replica21 --port 27021 --logpath /home/replica21/rep21.log --logappend --fork --bind_ip_all --replSet replicaset2 --shardsvr

mongod --dbpath /home/replica22 --port 27022 --logpath /home/replica22/rep22.log --logappend --fork --bind_ip_all --replSet replicaset2 --shardsvr

mongod --dbpath /home/replica23 --port 27023 --logpath /home/replica23/rep23.log --logappend --fork --bind_ip_all --replSet replicaset2 --shardsvr

mongod --dbpath /home/replica31 --port 27031 --logpath /home/replica31/rep31.log --logappend --fork --bind_ip_all --replSet replicaset3 --shardsvr

mongod --dbpath /home/replica32 --port 27032 --logpath /home/replica32/rep32.log --logappend --fork --bind_ip_all --replSet replicaset3 --shardsvr

mongod --dbpath /home/replica33 --port 27033 --logpath /home/replica33/rep33.log --logappend --fork --bind_ip_all --replSet replicaset3 --shardsvr


2. connect to each first mongo instance with a port like 27011,27021, 27031

mongo --port 27011

rs.initiate()


output:

rs.initiate()

{

"info2" : "no configuration specified. Using a default configuration for the set",

"me" : "mysql8.localdomain:27011",

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1603458252, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1603458252, 1)

}

replicaset1:OTHER> 

replicaset1:PRIMARY> 


3. Add the members to the replica set 

rs.add("mysql8.localdomain:27012")

rs.add("mysql8.localdomain:27013")


output:

replicaset1:PRIMARY> rs.add("mysql8.localdomain:27012")

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1603458404, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1603458404, 1)

}

replicaset1:PRIMARY> rs.add("mysql8.localdomain:27013")

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1603458416, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

},

"operationTime" : Timestamp(1603458416, 1)

}


4. check the configuration 

rs.conf()


output:

replicaset1:PRIMARY> rs.conf()

{

"_id" : "replicaset1",

"version" : 3,

"protocolVersion" : NumberLong(1),

"writeConcernMajorityJournalDefault" : true,

"members" : [

{

"_id" : 0,

"host" : "mysql8.localdomain:27011",

"arbiterOnly" : false,

"buildIndexes" : true,

"hidden" : false,

"priority" : 1,

"tags" : {

},

"slaveDelay" : NumberLong(0),

"votes" : 1

},

{

"_id" : 1,

"host" : "mysql8.localdomain:27012",

"arbiterOnly" : false,

"buildIndexes" : true,

"hidden" : false,

"priority" : 1,

"tags" : {

},

"slaveDelay" : NumberLong(0),

"votes" : 1

},

{

"_id" : 2,

"host" : "mysql8.localdomain:27013",

"arbiterOnly" : false,

"buildIndexes" : true,

"hidden" : false,

"priority" : 1,

"tags" : {

},

"slaveDelay" : NumberLong(0),

"votes" : 1

}

],

"settings" : {

"chainingAllowed" : true,

"heartbeatIntervalMillis" : 2000,

"heartbeatTimeoutSecs" : 10,

"electionTimeoutMillis" : 10000,

"catchUpTimeoutMillis" : -1,

"catchUpTakeoverDelayMillis" : 30000,

"getLastErrorModes" : {

},

"getLastErrorDefaults" : {

"w" : 1,

"wtimeout" : 0

},

"replicaSetId" : ObjectId("5f92d4cc9e732c176884e26c")

}

}


5. Repeat the steps from step-2 to step-4 for the other 2 replicats

mongo --port 27021

rs.initiate()

--- connect to each primary replicat mongo instance and add members to the replica set ----

rs.add("mysql8.localdomain:27022")

rs.add("mysql8.localdomain:27023")

rs.conf()


mongo --port 27031

rs.initiate()

rs.add("mysql8.localdomain:27032")

rs.add("mysql8.localdomain:27033")

rs.conf()


6. configure config servers

-- create required directories ---

midir /home/config1

midir /home/config2

midir /home/config3


-- start config servers ---

mongod --dbpath /home/config1 --port 27014 --logpath /home/config1/config1.log --logappend --fork --bind_ip_all --replSet config_replica --configsvr

mongod --dbpath /home/config2 --port 27015 --logpath /home/config2/config2.log --logappend --fork --bind_ip_all --replSet config_replica --configsvr

mongod --dbpath /home/config3 --port 27016 --logpath /home/config3/config3.log --logappend --fork --bind_ip_all --replSet config_replica --configsvr


mongo --port 27014

rs.initiate()

rs.add("mysql8.localdomain:27015")

rs.add("mysql8.localdomain:27016")

rs.conf()


7. Startup the mongos instnace

mongos --configdb "config_replica/mysql8.localdomain:27014,mysql8.localdomain:27015,mysql8.localdomain:27016" --port 27060 --bind_ip_all


8. Adding shard 

Step-1: connect to mongo shell to the mongos

mongo "mongodb://mysql8.localdomain:27014,mysql8.localdomain:27015,mysql8.localdomain:27016/myDB?replicaSet=config_replica"


Step-2: add replica set as a shard

Connect to the shard server and add the existing primary replicat set into it. 

sh.addShard('replicaset1/mysql8.localdomain:27011')

sh.addShard('replicaset2/mysql8.localdomain:27021')

sh.addShard('replicaset3/mysql8.localdomain:27031')


Step-3: check the status before adding a shard

sh.status()


output:

mongos> sh.status()

--- Sharding Status --- 

  sharding version: {

  "_id" : 1,

  "minCompatibleVersion" : 5,

  "currentVersion" : 6,

  "clusterId" : ObjectId("5f93ded2f74cee708d2898fa")

  }

  shards:   <-------------------------------- no shard members still

  active mongoses:

  autosplit:

        Currently enabled: yes

  balancer:

        Currently enabled:  yes  <-------------------- balancer is enabled by default 

        Currently running:  no

        Failed balancer rounds in last 5 attempts:  0

        Migration Results for the last 24 hours: 

        No recent migrations  <----------------------- no recent migrations since it's a new configuration

  databases:

        {  "_id" : "config",  "primary" : "config",  "partitioned" : true } 


-- adding the shard member ---


mongos> sh.addShard('replicaset1/mysql8.localdomain:27011')

{

"shardAdded" : "replicaset1",

"ok" : 1,

"operationTime" : Timestamp(1603528187, 7),

"$clusterTime" : {

"clusterTime" : Timestamp(1603528187, 7),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}


mongos> sh.addShard('replicaset2/mysql8.localdomain:27021')

{

"shardAdded" : "replicaset2",

"ok" : 1,

"operationTime" : Timestamp(1603528204, 4),

"$clusterTime" : {

"clusterTime" : Timestamp(1603528204, 4),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.addShard('replicaset3/mysql8.localdomain:27031')

{

"shardAdded" : "replicaset3",

"ok" : 1,

"operationTime" : Timestamp(1603528211, 5),

"$clusterTime" : {

"clusterTime" : Timestamp(1603528211, 5),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}


Step-4: check the shard member status post addition of Shard


mongos> sh.status()

--- Sharding Status --- 

  sharding version: {

  "_id" : 1,

  "minCompatibleVersion" : 5,

  "currentVersion" : 6,

  "clusterId" : ObjectId("5f93ded2f74cee708d2898fa")

  }

  shards: <---------------------------------------------------- shard members 

        {  "_id" : "replicaset1",  "host" : "replicaset1/mysql8.localdomain:27011,mysql8.localdomain:27012,mysql8.localdomain:27013",  "state" : 1 }

        {  "_id" : "replicaset2",  "host" : "replicaset2/mysql8.localdomain:27021,mysql8.localdomain:27022,mysql8.localdomain:27023",  "state" : 1 }

        {  "_id" : "replicaset3",  "host" : "replicaset3/mysql8.localdomain:27031,mysql8.localdomain:27032,mysql8.localdomain:27033",  "state" : 1 }

  active mongoses:

        "4.2.1" : 1

  autosplit:

        Currently enabled: yes

  balancer:

        Currently enabled:  yes

        Currently running:  no

        Failed balancer rounds in last 5 attempts:  0

        Migration Results for the last 24 hours: 

                No recent migrations

  databases:

        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }



9. Shard a collection

Step-1: connect mongo shell and enable shard and add shard on collection

mongo --port 27060 

use school

for(i=0;i<=10;i++){

db.class.insert({_id:i, name:"class_number"+i, faculty:"faculty"})};

db.class.find();


output:

mongos> use school

switched to db school

mongos> for(i=0;i<=10;i++){

... db.class.insert({_id:i, name:"class_number"+i, faculty:"faculty"})};

WriteResult({ "nInserted" : 1 })

mongos> db.class.find();

{ "_id" : 0, "name" : "class_number0", "faculty" : "faculty" }

{ "_id" : 1, "name" : "class_number1", "faculty" : "faculty" }

{ "_id" : 2, "name" : "class_number2", "faculty" : "faculty" }

{ "_id" : 3, "name" : "class_number3", "faculty" : "faculty" }

{ "_id" : 4, "name" : "class_number4", "faculty" : "faculty" }

{ "_id" : 5, "name" : "class_number5", "faculty" : "faculty" }

{ "_id" : 6, "name" : "class_number6", "faculty" : "faculty" }

{ "_id" : 7, "name" : "class_number7", "faculty" : "faculty" }

{ "_id" : 8, "name" : "class_number8", "faculty" : "faculty" }

{ "_id" : 9, "name" : "class_number9", "faculty" : "faculty" }

{ "_id" : 10, "name" : "class_number10", "faculty" : "faculty" }


--- enable Sharding at collection level ---

sh.enableSharding("school")

sh.shardCollection("school.class",{_id:1}, true);


output:

mongos> sh.enableSharding("school");

{

"ok" : 1,

"operationTime" : Timestamp(1603533211, 3),

"$clusterTime" : {

"clusterTime" : Timestamp(1603533211, 3),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}

mongos> sh.shardCollection("school.class",{_id:1}, true);

{

"collectionsharded" : "school.class",

"collectionUUID" : UUID("b1100999-9766-4644-9e70-1272f5d2ed53"),

"ok" : 1,

"operationTime" : Timestamp(1603533273, 10),

"$clusterTime" : {

"clusterTime" : Timestamp(1603533273, 10),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

}

}


--- check the status ---

mongos> sh.status()

--- Sharding Status --- 

  sharding version: {

  "_id" : 1,

  "minCompatibleVersion" : 5,

  "currentVersion" : 6,

  "clusterId" : ObjectId("5f93ded2f74cee708d2898fa")

  }

  shards:

        {  "_id" : "replicaset1",  "host" : "replicaset1/mysql8.localdomain:27011,mysql8.localdomain:27012,mysql8.localdomain:27013",  "state" : 1 }

        {  "_id" : "replicaset2",  "host" : "replicaset2/mysql8.localdomain:27021,mysql8.localdomain:27022,mysql8.localdomain:27023",  "state" : 1 }

        {  "_id" : "replicaset3",  "host" : "replicaset3/mysql8.localdomain:27031,mysql8.localdomain:27032,mysql8.localdomain:27033",  "state" : 1 }

  active mongoses:

        "4.2.1" : 1

  autosplit:

        Currently enabled: yes

  balancer:

        Currently enabled:  yes

        Currently running:  no

        Failed balancer rounds in last 5 attempts:  0

        Migration Results for the last 24 hours: 

                No recent migrations

  databases:

        {  "_id" : "books",  "primary" : "replicaset3",  "partitioned" : true,  "version" : {  "uuid" : UUID("b2288371-5ede-4301-80c6-88bcb68f6b88"),  "lastMod" : 1 } }

        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }

                config.system.sessions

                        shard key: { "_id" : 1 }

                        unique: false

                        balancing: true

                        chunks:

                                replicaset1 1

                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : replicaset1 Timestamp(1, 0) 

        {  "_id" : "library",  "primary" : "replicaset2",  "partitioned" : true,  "version" : {  "uuid" : UUID("ae0e2aba-ad6f-458d-abb3-748fc5817330"),  "lastMod" : 1 } }

                library.books

                        shard key: { "_id" : 1 }

                        unique: true

                        balancing: true

                        chunks:

                                replicaset2 1

                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : replicaset2 Timestamp(1, 0) 

        {  "_id" : "school",  "primary" : "replicaset3",  "partitioned" : true,  "version" : {  "uuid" : UUID("4ad6bb8b-02fc-4752-bcc2-1db0678cfc70"),  "lastMod" : 1 } }

                school.class   <------------------------------------- collection shard 

                        shard key: { "_id" : 1 }

                        unique: true

                        balancing: true

                        chunks:

                                replicaset3 1

                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : replicaset3 Timestamp(1, 0) 



--- let's insert more records into the collection to understand better the sharded chunks --------

mongo --port 27060 

use school

for(i=11;i<=5000;i++){

db.class.insert({_id:i, name:"class_number"+i, faculty:"faculty"})};

db.class.find();


--- check the collection shard status ----

db.class.stats();


--- let's test the scenario to access the records from the collections ---

Test-1: with shard key "_id"

db.class.find({_id:1555}).explain()

Test-2: without Shard key "name"

db.class.find({name:1555}).explain()

compare the above 2 result which will cleary shows the Test-1 access only 1 shard and the test-2 access all the 3 shards.


Comments

Post a Comment

Popular posts from this blog

Auditing in Oracle database

Unified Auditing: Oracle Database 12c Release 1 introduces a new auditing feature called Unified Auditing (12.1). Oracle Unified Auditing modifies the database's fundamental auditing capabilities. In earlier versions of Oracle, each component has its own audit trail. With unified auditing, all auditing is compiled into a single repository and view. Due to the fact that all audit data is now in a single location and a single format, there is a double simplification as a result. This data is made available in a consistent format in the UNIFIED AUDIT TRAIL data dictionary view, which is made possible by the unified audit trail, which is stored in a read-only table in the AUDSYS schema in the SYSAUX tablespace.  DBAs can build audit rules and then assign them to various users using Oracle 12c Unified Auditing, which has the same advantages as giving users varying system capabilities based on their responsibilities. The unified_audit_trail database will contain all of the audit records....
Read more

rs.stepDown() in MongoDB replication

It instructs the primary of the replica set to become a secondary. After the primary steps down,the eligible secondaries will hold an election for primary. Generally, the rs.stepDown() does not immediately step down the primary. If no electable secondaries are up to date with the primary, the primary waits up to secondaryCatchUpPeriodSecs (by default 10 seconds)  for a secondary to catch up. Once an electable secondary is available, the method steps down the priamry.  Once stepped down, the original primary becomes a secondary and is ineligible from becoming primary again for the remainder of time specified by stepDownSecs. The rs.stepDown() method attempts to terminate long-running user's operations that block the primary from stepping down, such as an index build, a write operation, or a map-reduce job. Syntax: rs.stepDown(stepDownSecs, secondaryCatchUpPeriodSecs)
Read more

Tuples in Python

 Tuples Basically Tuples in Python are very similar to Lists, but, Tuples are immutable which means that they can not be modified/changed. We generally use Tuples to make sure the values are not changed like, dates, Year, and days of the week or dates on a Calendar.  Creating a Tuple: >>> my_t1=(1,2,3) >>> my_t2=('one','two','three') >>> my_t1 (1, 2, 3) >>> my_t2 ('one', 'two', 'three') -- check the length of tupple: >>> len(my_t1) 3 >>> len(my_t2) 3 -- Tuples can have mixed objects as like Lists. >>> my_t3=(1,'two',3,'four') >>> my_t3 (1, 'two', 3, 'four') -- indexing with tuples  >>> my_t3=(1,'two',3,'four') >>> my_t3[2] 3 Immutable Tuples We can not modify the value. Here is the example: >>> my_t3=(1,'two',3,'four') >>> my_t3[2]=4 Traceback (most recent call last):  ...
Read more