CheckPoint – Fichier ELG corrompu (NULLNULL)

Log files can become corrupted when running debug of Check Point daemons on SecurePlatform / Gaia OS

Solution ID: sk52120
Product: Security Management, Security Gateway
Version: R75, R71, NGX R62, NGX R65, R70, NGX R61, NGX R60, R75.20, R75.40, R75.40VS, R75.45, R75.46
OS: SecurePlatform, SecurePlatform 2.6, Gaia
Platform / Model: All
Date Created: 15-oct.-2010
Last Modified: 18-févr.-2013

 

SYMPTOMS

  • When running debug of Check Point daemons (FWM, CPD, FWD , VPND, etc), the corresponding log files (.ELG) might become corrupted : when open the file, it is mostly filled with ‘NULL’ characters (dots), and only at the end of the file, there are some readable lines.

 

CAUSE

The problem happens due to the fact that two different mechanisms open the ELG files for ‘Write’ operation.

1) When the daemon is under debug, the log file (.ELG) will be rotated by internal TDERROR mechanism.

2) However, there is another mechanism – a Task ‘RotateLogs’ registered under CPD daemon, which is called every 100 sec to rotate a log file if it reached the maximal configured size.

[Expert@FW]# cpd_sched_config print

Task: « RotateLogs »

Command: /sbin/cp_logrotate

Arguments:

Interval: 100

Active: true

RunAtStart: true

Most probable reason that today there are two different mechanisms for log rotation is due to historical reasons – the TDERROR mechanism was added later in time…

SOLUTION

Introduction:

Before collecting the debugs of Check Point daemons (FWM, CPD, FWD, VPND, etc), user needs to disable the rotation of the relevant log files (.ELG) through ‘/sbin/cp_logrotate’ command from Task ‘RotateLogs’, which is registered under CPD daemon, so it would not access the log files.

The Task ‘RotateLogs’ registered under CPD daemon is controlled by ‘/bin/log_start‘ shell script.

All the settings for various log files are stored in ‘/etc/cpshell/log_rotation.conf‘ file.

Important Note: starting in R75.20, the most imortant Check Point daemons were removed from this rotation – FWM, CPD, FWD, VPND.

 

Background:

It is possible to control the rotation of the relevant ELG log files in two ways:

  • Either run the ‘/bin/log_start’ script, which edits the ‘/etc/cpshell/log_rotation.conf’ file (preferred way).
  • Or edit the ‘/etc/cpshell/log_rotation.conf’ file manually.

This is the general syntax:

[Expert@HostName]# /bin/log_start

Usage: log –help

log list

log limit <log-index> <max-size> <backlog-copies>

log unlimit

log show <log-index> [<lines>]

[Expert@HostName]#

 

This is the default list of log files and their parameters (on versions lower than R75.20):

[Expert@HostName]# /bin/log_start   list
 Index File                                     Max-Size  Back-logs
    0) messages                                    65536          4
    1) routing_messages                            64536          4
    2) wtmp                                        65536          4
    3) lastlog                                    262400          4
    4) secure                                      64536          4
    5) cpstart.log                               1048576          4
    6) CPbackup.elg                                64536          4
    7) backup_logs.elg                             64536          4
    8) fwd.elg                                     64536          4
    9) dtlsd.elg                                 1048576          4
   10) dtpsd.elg                                 1048576          4
   11) sdsd.elg                                  1048576          4
   12) vpnd.elg                                  1048576          4
   13) aftpd.elg                                 1048576          4
   14) atelnetd.elg                              1048576          4
   15) ahttpd.elg                                1048576          4
   16) unifiedd.elg                              1048576          4
   17) aclientd.elg                              1048576          4
   18) ahclientd.elg                             1048576          4
   19) arlogind.elg                              1048576          4
   20) asessiond.elg                             1048576          4
   21) asmtpd.elg                                1048576          4
   22) aufpd.elg                                 1048576          4
   23) genericd.elg                              1048576          4
   24) lhttpd.elg                                1048576          4
   25) pingd.elg                                 1048576          4
   26) mdq.elg                                   1048576          4
   27) snauth.elg                                1048576          4
   28) cp_http_server.elg                        1048576          4
   29) cpwd.elg                                  1048576          4
   30) cpd.elg                                   1048576          4
   31) cpwd.elg                                  1048576          4
   32) stormc.elg                                1048576          4
   33) cprid.elg                                 1048576          4
   34) rtmd.elg                                  1048576          4
   35) fwm.elg                                   1048576          4
   36) fgd50.elg                                 1048576          4
   37) boot.log                                  1048576          4
[Expert@HostName]#

This is the default content of configuration file (on versions lower than R75.20):

[Expert@HostName]# cat   /etc/cpshell/log_rotation.conf
/var/log/messages                        65536          4
/var/log/routing_messages                64536          4
/var/log/wtmp                            65536          4
/var/log/lastlog                        262400          4
/var/log/secure                          64536          4
$CPDIR/log/cpstart.log                 1048576          4
/var/log/CPbackup.elg                    64536          4
/var/CPbackup/log/backup_logs.elg        64536          4
$FWDIR/log/fwd.elg                     1048576          4
$FWDIR/log/dtlsd.elg                   1048576          4
$FWDIR/log/dtpsd.elg                   1048576          4
$FWDIR/log/sdsd.elg                    1048576          4
$FWDIR/log/vpnd.elg                    1048576          4
$FWDIR/log/aftpd.elg                   1048576          4
$FWDIR/log/atelnetd.elg                1048576          4
$FWDIR/log/ahttpd.elg                  1048576          4
$FWDIR/log/unifiedd.elg                1048576          4
$FWDIR/log/aclientd.elg                1048576          4
$FWDIR/log/ahclientd.elg               1048576          4
$FWDIR/log/arlogind.elg                1048576          4
$FWDIR/log/asessiond.elg               1048576          4
$FWDIR/log/asmtpd.elg                  1048576          4
$FWDIR/log/aufpd.elg                   1048576          4
$FWDIR/log/genericd.elg                1048576          4
$FWDIR/log/lhttpd.elg                  1048576          4
$FWDIR/log/pingd.elg                   1048576          4
$FWDIR/log/mdq.elg                     1048576         10
$FWDIR/log/snauth.elg                  1048576          4
$CPDIR/log/cp_http_server.elg          1048576          4
$CPDIR/log/cpwd.elg                    1048576          4
$CPDIR/log/cpd.elg                     1048576          4
$CPDIR/log/cpwd.elg                    1048576          4
$FWDIR/log/stormc.elg                  1048576          4
$CPDIR/log/cprid.elg                   1048576          4
$FWDIR/log/rtmd.elg                    1048576          4
$FWDIR/log/fwm.elg                     1048576          4
$FGDIR/log/fgd50.elg                   1048576          4
/var/log/boot.log                      1048576          4
[Expert@HostName]#

 

Action plan:

(Step 1) Backup the configuration file:

[Expert@HostName]# cp /etc/cpshell/log_rotation.conf /etc/cpshell/log_rotation.conf_BACKUP

(Step 2) Find the index of the relevant logs in the configuration file:

[Expert@HostName]# /bin/log_start list

Example:

[Expert@FW]# /bin/log_start   list | grep -E "fwd|cpd|fwm"
8) fwd.elg                                    1048576          4
30) cpd.elg                                   1048576          4
35) fwm.elg                                   1048576          4

 

(Step 3) Disable the rotation of each relevant log:

 

[Expert@HostName]# /bin/log_start   unlimit   LOG-INDEX

(run ‘/bin/log_start list‘ to verify)

 

Example for CPD.elg:

[Expert@HostName]# /bin/log_start   list
....................
30) cpd.elg                                   1048576          4
....................

 

[Expert@HostName]# /bin/log_start   unlimit   30

 

[Expert@HostName]# /bin/log_start   list
....................
30) cpd.elg                                   unlimited          unlimited
....................

 

Notes about ‘/bin/log_start unlimit LOG-INDEX‘ command:

  • applies immediately
  • does not require any additional actions (no need to ‘cpstop;cpstart‘ or ‘reboot‘)
  • survives ‘cpstop;cpstart
  • survives reboot

 

(Step 4) Collect the necessary debugs of the relevant daemons with TDERROR:

(A) Start:

[Expert@HostName]# fw  debug  fwm  on  TDERROR_ALL_ALL=5
[Expert@HostName]# cpd_admin  debug  on  TDERROR_ALL_ALL=5
[Expert@HostName]# fw  debug  fwd  on  TDERROR_ALL_ALL=5

(B) Replicate the problem.

(C) Stop:

[Expert@HostName]# fw  debug  fwm  off  TDERROR_ALL_ALL=0
[Expert@HostName]# cpd_admin  debug  off  TDERROR_ALL_ALL=0
[Expert@HostName]# fw debug  fwd  off  TDERROR_ALL_ALL=0

(D) Collect the ELG log files.

 

Note:
When the log rotation by SPLAT OS is disabled via ‘/bin/log_start unlimit‘ command,
the log files will be rotated by TDERROR mechanism :
— when the log size reaches 20 MB, it will be rotated
— maximal number of log files will be 10

These numbers are hard-coded and can not be changed.

 

(Step 5) Restore the original settings for each log we changed:

(A) either with ‘/bin/log_start limit‘ command:

 

[Expert@HostName]# /bin/log_start   limit   LOG-INDEX   MAX-SIZE   BACKLOG-COPIES

(run ‘/bin/log_start list‘ to verify)

 

Example for CPD.elg:

[Expert@HostName]# /bin/log_start   list
....................
30) cpd.elg                                   unlimited          unlimited
....................

 

[Expert@HostName]# /bin/log_start   limit   30   1048576   4

 

[Expert@HostName]# /bin/log_start   list
....................
30) cpd.elg                                   1048576          4
....................

 

Notes about ‘/bin/log_start limit LOG-INDEX‘ command :

  • applies immediately
  • does not require any additional actions (no need to ‘cpstop;cpstart‘ or ‘reboot‘)
  • survives ‘cpstop;cpstart
  • survives reboot

 

(B) or by simply restoring the original configuration file:

 

[Expert@HostName]# cp -f   /etc/cpshell/log_rotation.conf_BACKUP   /etc/cpshell/log_rotation.conf

Laisser un commentaire

Entrez vos coordonnées ci-dessous ou cliquez sur une icône pour vous connecter:

Logo WordPress.com

Vous commentez à l'aide de votre compte WordPress.com. Déconnexion / Changer )

Image Twitter

Vous commentez à l'aide de votre compte Twitter. Déconnexion / Changer )

Photo Facebook

Vous commentez à l'aide de votre compte Facebook. Déconnexion / Changer )

Photo Google+

Vous commentez à l'aide de votre compte Google+. Déconnexion / Changer )

Connexion à %s