How to debug in kernel mode

Here is a list of the FireWall-1 Kernel Debug Options

The FireWall-1 kernel module may be put into debug mode with the following command:

nokia_fw[admin]# fw ctl debug

The following are valid debug-options         0 - turn debugging off (fw ctl debug 0)         all (DO NOT USE)              all debug features              The output is excessive, making the system unresponsive. Often, only a cold reboot will restore             access to the system.      cookie - "cookie" (abstract data type of representing packets) related messages  crypt - encryption related information  domain - domain queries  driver - device driver operations  filter - filter loading and unloading  hold - packets held and released (related, among other things, to encryption)  if - interface binding  install - driver installation  ioctl - ioctl commands from the daemon  kbuf - kernel buffers (buffers allocated by the kernel for encryption purposes)  ld - operations on dynamic tables  log - log messages sent to the daemon  machine - virtual machine operation (the virtual machine which executes the INSPECT code compiled from .pf files)  memory - memory usage  misc - all others  packet - packet handling  profile - performance monitoring  q - streams and queues operations  synatk - operations related to syn attack protection  tcpseq - TCP sequence numbers changed  winnt - windows NT specific operations  xlate - address translation for new connections  xltrc - address translation for telnet and ftp

Redirecting Output to a File

The information is sent by default to the console. It can also be sent to a kernel buffer. This is necessary because the output often is to great to process real-time. Here are some examples of how to redirect the output to a file for examination later:

tla[admin]# fw ctl debug -buf [1024]

The default size is 1024 Kbytes. At this point, you have only enabled the redirection of stdout to a buffer, but the next step is retrieve the contents of this buffer. This is down with the following command

tla[admin]# fw ctl kdebug -f

This will now dump the buffer to stdout, but this is similar to before. The following are the steps to redirect the buffer to a file:

tla[admin]# fw ctl debug -buf

tla[admin]# fw ctl debug

tla[admin]# fw ctl kdebug -f > filespec

tla[admin]# tail -f filespec

When you have gathered enough information, press to stop the output to the file. You will have to issue `fw ctl debug 0` in order to actually restore the kernel to normal operation.

Debugging HTTP Security Server

We used these below when we debugged HTTP Security Server problems. One of the vulnerabilities in the HTTP Security Server is that it will block all network connections it is checking if a URL is not resolvable. This is serious in that a DOS of DNS to your firewall can cripple it. For example, if you create a URI resource object to explicitly block HTTP to www.somedomain.com and this does not resolve to an IP address, then all HTTP that is subject to Content Security will be blocked.

tla[admin]# setenv HTTP_DEBUG 1

tla[admin]# setenv FWAHTTPD_DEBUG 1

tla[admin]# setenv FW_DEBUG_EVENT 1

tla[admin]# setenv FWT_DEBUG all

tla[admin]# fw kill fwd; fwd `cat $FWDIR/conf/masters`

The later versions of FireWall-1 enable SMTP_DEBUG and MDQ_DEBUG in another way. These variables should be defined in the $FWDIR/conf/smtp.conf file and then the fwd process should be killed using the –USR1 switch, when this is done, the debugging information will start immediately without the need to restart the daemons. To remove these environmental variables, execute unsetenv env_variable The output is directed to $FWDIR/log/ahttpd.log. This particular problem produced numerous duplicate entries in the log file that were of this form:
[port 1023:Connection refused Thu Aug 5 23:42:57 1999] [pid=764] Failed to connect to server for side = 1 at [Thu Aug 5 23:42:57 1999] [pid=764] write_from_queue : side = 1, clnt = 0,
buf = (
:data (
:resolved_name (www.unresolveable.com) :type (dns_resolve_byname)
) :chain_name (resolver_list) :call_function (cached_resolver_gethostbyname) :return_function () :serial_number_resolver_list0(2) :current_side(1)
)
The speculation was that FireWall-1 was attempting, over and over, to resolve to an IP address. It was verified that this particular destination was not resolvable. Once the rule using a URI resource object of type Wildcard, which explicitly specified this site, was removed, everything was restored. This bug was verified to be in 4.0 SP4 for Solaris on Aug 6th, 1999. The immediate solution is to not use a URI resource object of type Wildcard to drop or reject HTTP, but to only Accept HTTP.

Debugging SMTP Security Server

We use the following to debug SMTP Security Server. At this point in time, we do not have a good definition of what these variables do with the exception that they all increase the output of debug information. The variables with MDQ put the spool dequeuer process into debug mode. The SMTP_DEBUG environmental variable is shown with three levels. Choose one. FWT_DEBUG is associated with the fwd daemon. OPSEC_DEBUG_LEVEL

tla[admin]# setenv MDQ_DEBUG 1

tla[admin]# setenv FWMDQ_DEBUG 1

tla[admin]# setenv SMTP_DEBUG [1 2 3]

tla[admin]# setenv FWD_DEBUG cvp

tla[admin]# setenv FWT_DEBUG cvp

tla[admin]# setenv OPSEC_DEBUG_LEVEL [1 2 3]

tla[admin]# fw kill fwd;fwd `cat $FWDIR/conf/masters`

To remove these environmental variables, execute unsetenv env_variable.

Debugging SecuRemote/Encapsulation problem.

tla[admin]# fw ctl debug cookie

tla[admin]# fw ctl debug -buf 100

tla[admin]# fw ctl kdebug -f filespec

tla[admin]# tail -f filespec

 

We should see messages of the form "cookie data could not XXX". There will be messages that specifically complain about fragmentation. (SecuRemote may be placed into debug mode by creating the file, fwenc.log, at the root of your system drive. For example, this might be c:\fwenc.log)

 Debugging the in.pingd daemon

 

tla[admin]# setenv FWPING_DEBUG 1

 

The output of fw tab -t check_alive is also analyzed