Guru Meditation Reports

A Guru Meditation Report (GMR) is generated by the Designate services when service processes receiving SIGUSR2 signal. The report is a general-purpose debug report for developers and system admins which contains the current state of a running Designate service process.

Structure of a GMR

Package
Shows information about the package to which this process belongs, including version information
Threads
Shows stack traces and thread ids for each of the threads within this process
Green Threads
Shows stack traces for each of the green threads within this process (green threads don’t have thread ids)
Processes
Shows information about this process, including pid, ppid, uid and process state
Configuration
Lists all the configuration options currently accessible via the CONF object for the current process

Generate a GMR

A GMR can be generated by sending the USR2 signal to any Designate processes.

For example, suppose designate-central has pid 15097, kill -USR2 15097 will trigger a GMR.

If option logdir has been set in designate.conf, the GMR will be saved in the folder which logdir specified. Otherwise, the GMR will be printed to the stderr.

Reference

For more information about GMR, see GMR wiki.

GMR Example

========================================================================
====                        Guru Meditation                         ====
========================================================================
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||


========================================================================
====                            Package                             ====
========================================================================
product = OpenStack Designate
vendor = OpenStack Foundation
version = 2015.1
========================================================================
====                            Threads                             ====
========================================================================
------                  Thread #140098874533632                   ------

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py:346 in run
    `self.wait(sleep_time)`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py:85 in wait
    `presult = self.do_poll(seconds)`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/epolls.py:62 in do_poll
    `return self.poll.poll(seconds)`

========================================================================
====                         Green Threads                          ====
========================================================================
------                        Green Thread                        ------

/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py:214 in main
    `result = function(*args, **kwargs)`

/opt/stack/designate/designate/openstack/common/service.py:492 in run_service
    `done.wait()`

/usr/local/lib/python2.7/dist-packages/eventlet/event.py:121 in wait
    `return hubs.get_hub().switch()`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py:294 in switch
    `return self.greenlet.switch()`

------                        Green Thread                        ------

/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py:214 in main
    `result = function(*args, **kwargs)`

/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py:95 in inner_func
    `return infunc(*args, **kwargs)`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_executors/impl_eventlet.py:96 in _executor_thread
    `incoming = self.listener.poll()`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:121 in poll
    `self.conn.consume(limit=1, timeout=timeout)`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:867 in consume
    `six.next(it)`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:782 in iterconsume
    `yield self.ensure(_error_callback, _consume)`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:688 in ensure
    `ret, channel = autoretry_method()`

/usr/local/lib/python2.7/dist-packages/kombu/connection.py:436 in _ensured
    `return fun(*args, **kwargs)`

/usr/local/lib/python2.7/dist-packages/kombu/connection.py:508 in __call__
    `return fun(*args, channel=channels[0], **kwargs), channels[0]`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:675 in execute_method
    `method()`

/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:774 in _consume
    `return self.connection.drain_events(timeout=poll_timeout)`

/usr/local/lib/python2.7/dist-packages/kombu/connection.py:275 in drain_events
    `return self.transport.drain_events(self.connection, **kwargs)`

/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py:91 in drain_events
    `return connection.drain_events(**kwargs)`

/usr/local/lib/python2.7/dist-packages/amqp/connection.py:302 in drain_events
    `chanmap, None, timeout=timeout,`

/usr/local/lib/python2.7/dist-packages/amqp/connection.py:365 in _wait_multiple
    `channel, method_sig, args, content = read_timeout(timeout)`

/usr/local/lib/python2.7/dist-packages/amqp/connection.py:336 in read_timeout
    `return self.method_reader.read_method()`

/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py:186 in read_method
    `self._next_method()`

/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py:107 in _next_method
    `frame_type, channel, payload = read_frame()`

/usr/local/lib/python2.7/dist-packages/amqp/transport.py:154 in read_frame
    `frame_header = read(7, True)`

/usr/local/lib/python2.7/dist-packages/amqp/transport.py:277 in _read
    `s = recv(n - len(rbuf))`

/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py:326 in recv
    `timeout_exc=socket.timeout("timed out"))`

/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py:201 in _trampoline
    `mark_as_closed=self._mark_as_closed)`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/__init__.py:162 in trampoline
    `return hub.switch()`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py:294 in switch
    `return self.greenlet.switch()`

------                        Green Thread                        ------

/usr/local/bin/designate-central:10 in <module>
    `sys.exit(main())`

/opt/stack/designate/designate/cmd/central.py:37 in main
    `service.wait()`

/opt/stack/designate/designate/service.py:356 in wait
    `_launcher.wait()`

/opt/stack/designate/designate/openstack/common/service.py:187 in wait
    `status, signo = self._wait_for_exit_or_signal(ready_callback)`

/opt/stack/designate/designate/openstack/common/service.py:170 in _wait_for_exit_or_signal
    `super(ServiceLauncher, self).wait()`

/opt/stack/designate/designate/openstack/common/service.py:133 in wait
    `self.services.wait()`

/opt/stack/designate/designate/openstack/common/service.py:473 in wait
    `self.tg.wait()`

/opt/stack/designate/designate/openstack/common/threadgroup.py:145 in wait
    `x.wait()`

/opt/stack/designate/designate/openstack/common/threadgroup.py:47 in wait
    `return self.thread.wait()`

/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py:175 in wait
    `return self._exit_event.wait()`

/usr/local/lib/python2.7/dist-packages/eventlet/event.py:121 in wait
    `return hubs.get_hub().switch()`

/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py:294 in switch
    `return self.greenlet.switch()`

------                        Green Thread                        ------

No Traceback!

========================================================================
====                           Processes                            ====
========================================================================
Process 15097 (under 7312) [ run by: stanzgy (1000), state: running ]

========================================================================
====                         Configuration                          ====
========================================================================

backend:agent:bind9:
  query-destination = 127.0.0.1
  rndc-config-file = None
  rndc-host = 127.0.0.1
  rndc-key-file = None
  rndc-port = 953
  zone-file-path = /opt/stack/data/designate/zones

backend:bind9:
  masters =
    127.0.0.1:5354
  rndc-config-file = None
  rndc-host = 127.0.0.1
  rndc-key-file = None
  rndc-port = 953
  server_ids =

backend:fake:
  masters =
    127.0.0.1:5354
  server_ids =

backend:powerdns:
  backend = sqlalchemy
  connection = ***
  connection_debug = 0
  connection_trace = False
  db_inc_retry_interval = True
  db_max_retries = 20
  db_max_retry_interval = 10
  db_retry_interval = 1
  idle_timeout = 3600
  masters =
    10.180.64.117:5354
  max_overflow = None
  max_pool_size = None
  max_retries = 10
  min_pool_size = 1
  mysql_sql_mode = TRADITIONAL
  pool_timeout = None
  retry_interval = 10
  server_ids =
    f26e0b32-736f-4f0a-831b-039a415c481e
  slave_connection = ***
  sqlite_db = oslo.sqlite
  sqlite_synchronous = True
  use_db_reconnect = False

backend:powerdns:f26e0b32-736f-4f0a-831b-039a415c481e:
  backend = None
  connection = ***
  connection_debug = None
  connection_trace = None
  db_inc_retry_interval = None
  db_max_retries = None
  db_max_retry_interval = None
  db_retry_interval = None
  host = 10.180.64.117
  idle_timeout = None
  masters = None
  max_overflow = None
  max_pool_size = None
  max_retries = None
  min_pool_size = None
  mysql_sql_mode = None
  pool_timeout = None
  port = 53
  retry_interval = None
  slave_connection = ***
  sqlite_db = None
  sqlite_synchronous = None
  tsig-key = None
  use_db_reconnect = None

default:
  allowed_remote_exmods =
  backdoor_port = None
  backlog = 4096
  central-topic = central
  config-dir = None
  config-file =
    /etc/designate/designate.conf
  control_exchange = designate
  debug = True
  default-soa-expire = 86400
  default-soa-minimum = 3600
  default-soa-refresh-min = 3500
  default-soa-refresh-max = 3600
  default-soa-retry = 600
  default-ttl = 3600
  default_log_levels =
    amqp=WARN
    amqplib=WARN
    boto=WARN
    eventlet.wsgi.server=WARN
    keystone=INFO
    keystonemiddleware.auth_token=INFO
    oslo.messaging=WARN
    sqlalchemy=WARN
    stevedore=WARN
    suds=INFO
  fatal_deprecations = False
  host = cns-dev2
  instance_format = [instance: %(uuid)s]
  instance_uuid_format = [instance: %(uuid)s]
  log-config-append = None
  log-date-format = %Y-%m-%d %H:%M:%S
  log-dir = /opt/stack/logs/designate
  log-file = None
  log-format = None
  logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user)s %(tenant)s%(color)s] %(instance)s%(color)s%(message)s
  logging_debug_format_suffix = from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d
  logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
  logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
  mdns-topic = mdns
  network_api = neutron
  notification_driver =
  notification_topics =
    notifications
  policy_default_rule = default
  policy_dirs =
    policy.d
  policy_file = /etc/designate/policy.json
  pool-manager-topic = pool_manager
  publish_errors = False
  pybasedir = /opt/stack/designate
  quota-domain-records = 500
  quota-domain-recordsets = 500
  quota-domains = 10
  quota-driver = storage
  quota-recordset-records = 20
  root-helper = sudo designate-rootwrap /etc/designate/rootwrap.conf
  rpc_backend = rabbit
  rpc_thread_pool_size = 64
  state-path = /opt/stack/data/designate
  syslog-log-facility = LOG_USER
  tcp_keepidle = 600
  transport_url = None
  use-syslog = False
  use-syslog-rfc-format = False
  use_stderr = True
  verbose = True

network_api:neutron:
  admin_password = ***
  admin_tenant_name = None
  admin_username = None
  auth_strategy = keystone
  auth_url = None
  ca_certificates_file = None
  endpoint_type = publicURL
  endpoints = None
  insecure = False
  timeout = 30

oslo_concurrency:
  disable_process_locking = False
  lock_path = None

oslo_messaging_rabbit:
  amqp_auto_delete = False
  amqp_durable_queues = False
  fake_rabbit = False
  kombu_reconnect_delay = 1.0
  kombu_ssl_ca_certs =
  kombu_ssl_certfile =
  kombu_ssl_keyfile =
  kombu_ssl_version =
  rabbit_ha_queues = False
  rabbit_host = localhost
  rabbit_hosts =
    127.0.0.1
  rabbit_login_method = AMQPLAIN
  rabbit_max_retries = 0
  rabbit_password = ***
  rabbit_port = 5672
  rabbit_retry_backoff = 2
  rabbit_retry_interval = 1
  rabbit_use_ssl = False
  rabbit_userid = stackrabbit
  rabbit_virtual_host = /
  rpc_conn_pool_size = 30

proxy:
  http_proxy = None
  https_proxy = None
  no_proxy =

service:central:
  default_pool_id = 794ccc2c-d751-44fe-b57f-8894c9f5c842
  enabled-notification-handlers =
  managed_resource_email = hostmaster@example.com
  managed_resource_tenant_id = None
  max_domain_name_len = 255
  max_recordset_name_len = 255
  min_ttl = None
  storage-driver = sqlalchemy
  workers = None

service:pool_manager:
  backends =
    powerdns
  cache-driver = sqlalchemy
  enable-recovery-timer = True
  enable-sync-timer = True
  periodic-recovery-interval = 120
  periodic-sync-interval = 300
  periodic-sync-seconds = None
  poll-delay = 1
  poll-max-retries = 3
  poll-retry-interval = 2
  poll-timeout = 30
  pool-id = 794ccc2c-d751-44fe-b57f-8894c9f5c842
  threshold-percentage = 100
  workers = None

ssl:
  ca_file = None
  cert_file = None
  key_file = None

storage:sqlalchemy:
  backend = sqlalchemy
  connection = ***
  connection_debug = 0
  connection_trace = False
  db_inc_retry_interval = True
  db_max_retries = 20
  db_max_retry_interval = 10
  db_retry_interval = 1
  idle_timeout = 3600
  max_overflow = None
  max_pool_size = None
  max_retries = 10
  min_pool_size = 1
  mysql_sql_mode = TRADITIONAL
  pool_timeout = None
  retry_interval = 10
  slave_connection = ***
  sqlite_db = oslo.sqlite
  sqlite_synchronous = True
  use_db_reconnect = False