Diagnosing Checkstops

This document discusses checkstops, a machine check that occurs during another machine check. This document applies to AIX Versions 3.2, 4.1, 4.2 and 4.3.

For more in-depth coverage of this subject, the following IBM documents are recommended:

• AIX Version 4.1 Software Problem Debugging and Reporting for the RISC System/6000 (GG24-2513)
• Common Diagnostics and Service Guide (SA23-2687)
• Diagnostic Information For Micro Channel Bus Systems (SA23-2765)
• Diagnostic Information for Multiple Bus Systems (SA38-0509)
• Problem Solving Guide and Reference (SA23-2204, SA23-2606)
• System Management Guide, V3.2 (SC23-2457)
• System Management Guide, V4 (SC23-2525)
• Hardware Technical Information General Architectures (SA23-2643)

The AIX and RS/6000 product documentation library is also available:

A checkstop is indicated by an LED value of 185, 186, or 187 on the LED display of the main unit. If the machine does not have an LED display or the machine has been rebooted, then evidence of a checkstop should exist in the system error report. Look for an entry labeled CHECKSTOP in the error report to determine if a checkstop occurred.

A machine check is an error logged by the machine check handler. Causes of a machine check could be:

• an un-correctable (double bit) memory error
• an access to a memory address that does not exist

A non-maskable interrupt (NMI) is generated. The operating system logs the machine check, including various error logging registers reporting the cause of the machine check, and a system dump initiates.

A checkstop is a machine check that occurs during another machine check. A checkstop also occurs when the machine (usually a processor but sometimes a cache, memory, or I/O bus controller) determines that something is in an "impossible" state. An error occurs that cannot be isolated to a particular bus transfer in progress, or a processor detects no progress being made. The processor is not able to complete any instructions for some period of time.

When a system checkstops, the clocks in the machine are frozen within a few cycles after the error and the service processor saves the part of the state of the CPUs in NVRAM. It then attempts to do a full hardware reset and restart the system a number of times.

When the system reboots, the data is copied to a file in the /usr/lib/ras directory (ras stands for reliability and service). Two file names are used (checkstop.A and checkstop.B) in a rotating manner. The total number of checkstops that occurred during the reboot attempts, before the system came up successfully, is logged in the error log entry along with the file name.

If a second machine check occurs before the operating system completes logging the error (to NVRAM) and initiates a complete hardware reset or halts, the processor will checkstop.

Checkstops are inherently hardware phenomena. They do not necessarily indicate a solid failure of a component, so diagnostics will rarely determine that a problem exists. The checkstop file that is generated is required to determine the cause of the checkstop and what corrective actions are needed to resolve the situation. This file would be examined by your hardware service organization. For further information, contact one of the following:

• your local branch office
• your point of sale
• your hardware service organization

Use the following instructions to package these files for hardware service examination.

Gather system information by performing the following steps:

1. Run the command snap -g. This will put general information about the machine in the directory /tmp/ibmsupt.

2. Copy the checkstop files in /usr/lib/ras to the directory /tmp/ibmsupt/testcase. Execute:

      cp /usr/lib/ras/checkstop* /tmp/ibmsupt/testcase 
   

3. Make a file called customer_info in the directory /tmp/ibmsupt/other. In this file, include the following information:
   main contact
   telephone number of main contact
   machine type having the problem (examples: 7011, 7012, 7015)
   serial number of machine
   location of machine (physical location of machine including address)
   description of activity of machine prior to event

4. Put the testcase on diskette:

      tar -cvf /dev/fd0 /tmp/ibmsupt 
   

   fd0 is the floppy device.

5. Label the tape or diskette with the following information:
   customer name
   customer number
   incident#
   the command used to copy the information to tape

   Very Important: If the person sending in this testcase is not the person who reported the problem, be sure to include the name of the person who reported it. If the proper information is not on the package, then it takes valuable time to process and delays solving your problem. The incident# will be the reference number that your hardware service organization assigns to this problem.

AIX Version 3.2

AIX Version 4.1

AIX 4.2

AIX 4.3