Detailed Explanation of Lease Mechanism and Its Application in HBase

Detailed Explanation of Lease Mechanism and Its Application in HBase

Why is Lease Needed

Why does a distributed system need a lease mechanism? This is because in a distributed system, to ensure the high availability of the service, it is necessary to start another service instance in time to replace the failed service when a service fails. This requires maintaining a heartbeat information between the server and the client or between the server and the control center, which is used for the service process to report its current health status to the control center. If the control center does not receive the heartbeat report from the service process for a period of time, it will start a new process to continue providing services to the outside.

However, due to the complexity of the actual network situation, when the control center does not receive heartbeats, it cannot accurately determine whether the service has failed or whether there is a network failure between the service process and the control center. In this case, the control center's hasty activation of a new process may cause a

To avoid the occurrence of the above situation, a lease mechanism is introduced. At this time, the service node continuously applies for short-term leases to the control center. Before the expired time of the dispatched leases, the control center will not enable a new service node. If the service node's lease expires and it still cannot obtain a new lease from the control center, it will interrupt the client connection itself.

In addition, the lease mechanism can also be used for decoupling between the client and the server, to avoid the long-term occupation of server resources by the client process when it loses responsiveness, which may affect the stability of the server.

Implementation of Lease

In practical systems, if there is a significant risk in relying on a central node to publish leases outwardly, that is, if the central node fails or a network failure occurs, then the service nodes, due to not receiving new leases, will lead to the entire service cluster entering an unavailable state. Therefore, in actual use, the lease service provided to the outside is often composed of another set of clusters consisting of multiple process instances, which has high availability and can provide lease services to the outside, such as a zookeeper cluster.

Lease management of HRegionServer's lease

Initialization of lease thread

In the main loop of HRegionServer, preRegistrationInitialization is called to pre-initialize some threads, including initializing cluster connection information setupClusterConnection(), healthCheckChore, pauseMonitor, initializeZookeeper, and initializeThreads().

Among them, the initializeThreads() method initializes various threads, including the lease thread of this regionServer:

this.compactionChecker = new CompactionChecker(this, this.threadWakeFrequency, this); //Check merge requests 
this.periodicFlusher = new PeriodicMemstoreFlusher(this.threadWakeFrequency, this);  //Periodically check the flush requests of memstore 
this.leases = new Leases(this.threadWakeFrequency); 

　The definition of the Leases class is as follows, which inherits the abstract class HasThread and defines the following main member variables:

public static final int MIN_WAIT_TIME = 100; 
private final Map<String, Lease> leases = new ConcurrentHashMap<String, Lease>(); 
protected final int leaseCheckFrequency; 
protected volatile boolean stopRequested = false; 

　Among the Map type member variables, 'leases' is responsible for managing the lease instances in the regionserver process of this regionserver. Let's see what variables are defined in the Lease class:

private final String leaseName; 
private final LeaseListener listener; 
private int leaseTimeoutPeriod; 
private long expirationTime; 

　leaseTimeoutPeriod is the lease time, expirationTime will be set to the system time plus leaseTimeoutPeriod when the lease is created, used to periodically calculate how long the lease has been used. If the lease exceeds the expiration time defined by leaseTimeoutPeriod, it will trigger an expired event. LeaseListener will listen to this event and call the leaseExpired method. Different types of leases will inherit the LeaseListener interface and implement their own leaseExpired method, as follows is the implementation of the scan lease for this method:

@Override 
public void leaseExpired() {    //Handle lease expiration 
  RegionScannerHolder rsh = scanners.remove(this.scannerName); 
  if (rsh != null) { 
   RegionScanner s = rsh.s; 
   LOG.info("扫描器 " + this.scannerName + "区域租约已过期 " 
     + s.getRegionInfo().getRegionNameAsString()); 
   try { 
    Region region = regionServer.getRegion(s.getRegionInfo().getRegionName()); 
    if (region != null && region.getCoprocessorHost() != null) { 
     region.getCoprocessorHost().preScannerClose(s); 
    } 
    s.close(); 
    if (region != null && region.getCoprocessorHost() != null) { 
     region.getCoprocessorHost().postScannerClose(s); 
    } 
   } catch (IOException e) { 
    LOG.error("关闭扫描器用于 " 
     + s.getRegionInfo().getRegionNameAsString(), e); 
   } 
  } else { 
   LOG.warn("Scanner " + this.scannerName + "租约已过期，但无相关" + 
    "scanner found, hence no chance to close that related scanner!") 
  } 
 } 

　The client's scan request is decomposed into multiple RPC requests sent to the server, the number of decompositions is the ratio of the total data volume of the scan to the client's setCache. After each scan request is sent to the server, a scanner is rented to be used after the current scan is completed, so that subsequent scans can directly reuse existing resources. However, to prevent scanners from occupying server resources for a long time, they are closed through lease management when they are no longer in use.

OK, let's go back to the Leases class and see how it manages the various leases in the regionServer process, this logic is in its overridden run method:

public void run() { 
  long toWait = leaseCheckFrequency; 
  Lease nextLease = null; 
  long nextLeaseDelay = Long.MAX_VALUE; 
  while (!stopRequested || (stopRequested && !leases.isEmpty()) ) { 
   //sleep for a while 
   nextLease = null; 
   nextLeaseDelay = Long.MAX_VALUE; 
   for (Iterator<Map.Entry<String, Lease>> it = leases.entrySet().iterator(); it.hasNext();) { 
    Map.Entry<String, Lease> entry = it.next(); 
    Lease lease = entry.getValue(); 
    long thisLeaseDelay = lease.getDelay(TimeUnit.MILLISECONDS); 
    if ( thisLeaseDelay > 0) { 
     if (nextLease == null || thisLeaseDelay < nextLeaseDelay) { 
      nextLease = lease; 
      nextLeaseDelay = thisLeaseDelay; 
     } 
    } else { 
     // 租约已过期。在从地图中移除之前，运行过期代码。 
     // since its presence in the map is used to determine if the lease still exists. 
     if (lease.getListener() == null) { 
      LOG.error("lease listener is null for lease ") + lease.getLeaseName()); 
     } else { 
      lease.getListener().leaseExpired(); 
     } 
     it.remove(); 
    } 
   } 
  } 
  close(); 
 } 

　We omit some exception handling. During the cycle of the while loop, we will iterate over the lease managed in the map one by one, calculate each lease's thisLeaseDelay to check if this lease has expired. The method to determine if a lease has expired is very simple, that is, to take the difference between the current time and the expirationTime defined in the lease. If the difference is less than 0, it means that the lease has expired, and then call the leaseExpired method defined in the lease, which is related to what we have discussed above. ThisLeaseDelay determines when the next lease check will occur, and the calculation of thisLeaseDelay is based on the selection of the shortest leaseDelay among all unexpired leases. Calculate the toWait time through thisLeaseDelay, which is used to determine the sleep time in front.

Thank you for reading and hoping to help everyone. Thank you for your support of this site!

Declaration: The content of this article is from the Internet, and the copyright belongs to the original author. The content is contributed and uploaded by Internet users spontaneously. This website does not own the copyright, does not edit the content artificially, and does not assume relevant legal liability. If you find any content suspected of copyright infringement, please send an email to: notice#oldtoolbag.com (When reporting via email, please replace # with @) to report violations, and provide relevant evidence. Once verified, this site will immediately delete the content suspected of infringement.