Monday, November 4, 2019

Namenode migration in CDH cluster

  • Stop Namenode when they in the standby mode.
  • Backup the `dfs.name.dir, dfs.namenode.name.dir` directory.
  • Make sure the backup is restored at the same location on the target Namenode host and permission preserved.
  • Usually the permission is `hdfs:hadoop` if in case of missing it.
  • There are 5 important Namenode setting that need to be inherited to the target Namenode
    1. `dfs.ha.automatic-failover.enabled` checked
    2. `NameNode Nameservice`, usually it is `nameservice1`. But however it could vary depends on the case. It is good to record it before hand.
    3. `Mount Points`, ditto as per above
    4. `Quorum-based Storage Journal name` ditto as per above
    5. Java Opts setting
  • Namenode migration will be resonating with failover controller as well. Make sure they are migrated at the same time, same node.
  • Once we are ready for all the backup and recorded information, then we are good to kick the tyre get rolling.
  • First of all, delete the "Namenode(Standby)" role and "Failover Controller" from the host.
  • Add new role > select "Namenode" and "Failover Controller" to the new host.
  • Make sure all above mentioned Namenode settings are in place.
  • Close your palms and pray when you hit the start button on both. Should you start with Failover Controller first, later followed by Namenode.
  • Now you are relief once Namenode and Failover Controller started, you should now followed with a series of services restart.
  • Do a rolling restart on Data nodes 
  • Do a rolling restart on Hive Metastore Server
  • Do a rolling restart on Hive server
  • Do a rolling restart on Node Manager
  • Do a rolling restart on Resource Manager
  • Do a rolling restart on Oozie
  • Do a rolling restart on Httpfs
  • Do a rolling restart on Journal Nodes
  • Lastly Namenode
  • Show is completed. The End

No comments: