TerracottaでHA構成を試してみた

サーバーアレイまでではなくてシンプルなActive-Passive構成でHAを試してみた。

参考資料

HAにするためのサンプル構成とかないかと探してみたけど、わりと情報が散乱しているのでこのへんを総合して構成してやらないといけなかった。

構成

名前と役割があってないのはスルーしてください。

        ┌──────┐
        │ tc-master1 │    TomcatTerracotta(client)が動いてる
        └──────┘
         ┌──┴──┐
┌──────┐┌──────┐
│ tc-worker1 ││ tc-worker3 │HA構成のTerracotta(server)
└──────┘└──────┘

ポイントの整理

  • serversセクションはサーバーとクライアントですべて揃えておかないといけない
    • でも、election-timeのところはかたっぽずつ変更したら問題なく設定変更できた
  • サーバー起動時に、自分がどのサーバーなのかを明示する
  • クライアント側は普通にやればいい
  • 切り替え中はブラウザからみた場合、接続待ち状態になる
    • 問答無用で接続が拒否されたりしない

サーバー側のconfig

[root@tc-worker1 ~]# cat /opt/terracotta-3.0.0/tc-config-srvr.xml 

こんな感じで、tc-worker1とtc-worker3の両方で同じにしておかないといけない。persistenceのところをコメントアウトしてるのはファイル出力をさせなくてよかったから。

<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
<!--   for HA           -->

  <servers>
    <server host="tc-worker1" name="Server1">
      <data>/opt/terracotta/server1-data</data>
      <l2-group-port>9530</l2-group-port>
<!--
      <dso>
        <persistence>
          <mode>permanent-store</mode>
        </persistence>
      </dso>
-->
    </server>
    <server host="tc-worker3" name="Server3">
      <data>/opt/terracotta/server2-data</data>
      <l2-group-port>9530</l2-group-port>
<!--
      <dso>
          <mode>permanent-store</mode>
        </persistence>
      </dso>
-->
    </server>
     <ha>
        <mode>networked-active-passive</mode>
        <networked-active-passive>
             <election-time>5</election-time>
        </networked-active-passive>
     </ha>
  </servers>
</tc:tc-config>

サーバー起動

serverセクションのname属性に指定した名前の特定と、設定ファイルを指定して起動。

[root@tc-worker3 ~]# /opt/terracotta-3.0.0/bin/start-tc-server.sh  -n Server3 -f /opt/terracotta-3.0.0/tc-config-srvr.xml
[root@tc-worker1 ~]# /opt/terracotta-3.0.0/bin/start-tc-server.sh  -n Server1 -f /opt/terracotta-3.0.0/tc-config-srvr.xml

クライアント側の設定

[root@tc-master1 tc]# cat /opt/terracotta-3.0.0/tc-config-clt_tomcat.xml

serversのところはサーバー側とあわせておく。

<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">

  <servers>

  <!-- The <update-check> element helps ensure that you're using the latest version of Terracotta DSO.
    Out-of-date versions trigger a message to log and standard output. -->

<!--
       <update-check>
         <enabled>true</enabled>
       </update-check>
-->
    <server host="tc-worker1" name="Server1">
      <data>/opt/terracotta/server1-data</data>
      <l2-group-port>9530</l2-group-port>
<!--
      <dso>
        <persistence>          <mode>permanent-store</mode>
        </persistence>
      </dso>
-->
    </server>
    <server host="tc-worker3" name="Server3">
      <data>/opt/terracotta/server2-data</data>
      <l2-group-port>9530</l2-group-port>
<!--
      <dso>
          <mode>permanent-store</mode>
        </persistence>
      </dso>
-->
    </server>
     <ha>
        <mode>networked-active-passive</mode>
        <networked-active-passive>
             <election-time>5</election-time>
        </networked-active-passive>
     </ha>
  </servers>
 
 <!-- <modules> is a section in <clients> that specifies the TIMs to be used by your Terracotta installation.
      In this example, the TIM for an application server called "my-app-sever", version 1.0.1, is included.
      The version of the TIM, 1.2.3, is also specified. This configuration assumes that the default repository,
      the modules directory in the Terracotta installation directory, is being used to store the TIM files. 
      For more on TIM repositories, see the Terracotta Configuration Guide and Reference. -->
       
  <clients>
     <modules>
        <module name="tim-tomcat-6.0" version="1.0.1" />
     </modules>
  </clients>

  <application>
    <dso>
       <instrumented-classes>
         <include>
           <class-expression>sessions.DummyCart</class-expression>
         </include>
       </instrumented-classes>

      <web-applications>
        <web-application>examples</web-application>
      </web-applications>
    </dso>
   </application>
</tc:tc-config>

HA構成で動いてるときのログ

先に起動したほうがアクティブになる。

アクティブ側
2009-08-02 21:02:37,390 INFO - Becoming State[ ACTIVE-COORDINATOR ]
2009-08-02 21:02:37,501 INFO - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.

パッシブ側のサーバーが起動されると、Active-Passiveのそれぞれにこんな感じのログがでる。

アクティブ側
2009-08-02 20:48:35,050 INFO - NodeID[tc-worker3:9510] joined the cluster
パッシブ側
2009-08-02 20:48:35,051 INFO - NodeID[tc-worker1:9510] joined the cluster
2009-08-02 20:48:35,052 INFO - Moved to State[ PASSIVE-UNINITIALIZED ]
2009-08-02 20:48:35,177 INFO - Moved to State[ PASSIVE-STANDBY ]

アクティブ側にネットワーク不通やプロセス死亡などのなんらかの理由で接続できなくなるとパッシブ側にこんなログがでてアクティブになってクライアントと再接続したことがわかる。

パッシブ側
2009-08-02 20:48:09,642 WARN - NodeID[tc-worker3:9510] left the cluster
2009-08-02 20:48:11,647 INFO - Becoming State[ ACTIVE-COORDINATOR ]
2009-08-02 20:48:11,654 INFO - Starting reconnect window: 120000 ms. Waiting for 1 clients to connect. 
2009-08-02 20:48:11,655 INFO - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.

不通になったノードを再起動させると、クラスタに参加したことがログにでる。

パッシブ側
2009-08-02 21:04:32,992 INFO - NodeID[tc-worker1:9510] joined the cluster
2009-08-02 21:04:33,015 INFO - Moved to State[ PASSIVE-UNINITIALIZED ]
2009-08-02 21:04:33,099 INFO - Moved to State[ PASSIVE-STANDBY ]

クライアントを起動させるとこんなログがでて、クラスタトポロジーに参加したことがわかる。

クライアント側
2009-08-02 21:10:45,882 [main] INFO com.tc.bundles.Resolver - Resolved TIM org.terracotta.modules:tim-tomcat-6.0:1.0.1 from /opt/terracotta-3.0.0/modules/org/terracotta/modules/tim-tomcat-6.0/1.0.1/tim-tomcat-6.0-1.0.1.jar2009-08-02 21:10:47,183 [main] INFO com.terracottatech.dso - Trying to get Cluster topology from http://tc-worker1:9510/groupinfo2009-08-02 21:10:47,270 [main] INFO com.terracottatech.dso - Trying to get L1 Reconnect Properties from http://tc-worker1:9510/l1reconnectproperties
2009-08-02 21:10:47,325 [main] INFO com.tc.net.core.TCComm - Comm Worker Threads NOT requested
2009-08-02 21:10:47,337 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Client - HealthChecker Started2009-08-02 21:10:47,356 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener started at /0:0:0:0:0:0:0:0:56143
2009-08-02 21:10:47,492 [Statistics Logger] INFO com.terracottatech.dso - memory free : 43507048
2009-08-02 21:10:47,492 [Statistics Logger] INFO com.terracottatech.dso - memory used : 21373592
2009-08-02 21:10:47,492 [Statistics Logger] INFO com.terracottatech.dso - memory max : 64880640
2009-08-02 21:10:47,714 [main] INFO com.terracottatech.dso - Statistics buffer: '/home/tfi/tc/statistics-192.168.131.58'.
2009-08-02 21:10:47,746 [main] INFO com.tc.statistics.StatisticRetrievalAction - "message monitor" statistic is not enabled. Please enable the property "tcm.monitor.delay" to collect this statistics.
2009-08-02 21:10:47,772 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: Copy
2009-08-02 21:10:47,772 [main] INFO com.tc.runtime.TCMemoryManagerImpl - GarbageCollector: MarkSweepCompact
2009-08-02 21:10:47,869 [L1Management JMX registration] INFO com.tc.management.L1Management - Terracotta JMX connector available at[service:jmx:terracotta://localhost]2009-08-02 21:10:47,945 [main] INFO com.tc.net.protocol.transport.ClientMessageTransport - ConnectionID(-1.ffffffffffffffffffffffffffffffff): Attaching new connection: com.tc.net.core.TCConnectionJDK14@1711861090: connected: true, closed: false local=192.168.131.58:56605 remote=192.168.131.55:9510 connect=[Sun Aug 02 21:10:47 JST 2009] idle=4ms [0 read, 0 write]2009-08-02 21:10:47,968 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Client - Health monitoring agent started for tc-worker1.in.techfirm.co.jp:9510
2009-08-02 21:10:47,970 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:56143)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Client - HealthCheckCallbackPort verification PASSED for tc-worker1.in.techfirm.co.jp:9510(callbackport: 9510)2009-08-02 21:10:47,971 [main] INFO com.tc.management.remote.protocol.terracotta.TunnelingEventHandler - Client JMX server ready; sending notification to L2 server2009-08-02 21:10:47,975 [WorkerThread(client_coordination_stage,0)] INFO com.tc.object.handshakemanager.ClientHandshakeManagerImpl - ClientID[0]: Connected: Unpausing from State[ PAUSED ] RemoteNode : GroupID[0]. Disconnect count : 12009-08-02 21:10:48,011 [WorkerThread(client_coordination_stage,0)] INFO com.tc.object.handshakemanager.ClientHandshakeManagerImpl - ClientID[0]: Received Handshake ack for this node :GroupID[0]2009-08-02 21:10:48,011 [main] INFO com.terracottatech.console - Connection successfully established to server at 192.168.131.55:95102009-08-02 21:10:48,011 [main] INFO com.terracottatech.dso - Connection successfully established to server at 192.168.131.55:9510