Ricardo's space - Your Number 1Tech Place

Before addressing the issue at hand it is best to get familiar with the System process and what makes it so unique in comparison.

System Process and Threads 

The System process (PID 8 in Windows 2000 and process ID 4 in Windows XP; Vista and Windows 2003) houses kernel-mode system threads. System threads have all the attributes and contexts and are similar to regular user-mode threads, however they run only in kernel-mode. System threads don't have a user process address space and thus must allocate any dynamic info from OS system memory heaps, such as a paged or nonpaged pool.

The System process is not bound to an executable image like any other process. It's existence serves OS threads for Windows subsystems and device drivers as well. So if you notice CPU Spikes around the corner with the System process it could well be a misbehaving device driver.

How can I identify the device driver that's causing the CPU Spikes?

To answer the above question we can use process explorer to help us identify the culprit. Process Explorer shows the threads running in a process by consulting the Threads page of the Process Properties dialog.

You should configure process explorer to download symbols from MSFT to get thread function names for further information on it's activity.

In the bellow screen we can map CPU activity to the file server device driver (Srv.sys) that respond to network I/O requests for file data on disk partitions shared to the network. If you press the Module button you can see details about the driver Srv.sys in this case. Because I have previously configured symbols for OS images in Process Explorer the thread list also showed function names in this case (Worker Thread) functions.

One way to look inside a thread execution is to look at the thread stack as bellow with Process Explorer. The stack is a memory region that stores function history invocation. To do it you only need to press the Stack button.

IMPORTANT! In Windows Vista we cannot get thread info from Process Explorer as we did above as Vista sees the System process as a protected process. The Windows Vista operating system introduces a new type of process, called a protected process, to enhance support for digital rights management functionality in Windows Vista. Protected processes exist alongside normal processes in Windows Vista. For more info please download this whitepaper from MSFT: http://www.microsoft.com/whdc/system/vista/process_vista.mspx

However if the thread running is one of the system worker threads we still don't really know what the thread is doing because any device driver can submit work to a system worker thread.

Another way to see what the Worker threads were doing is to turn to Kernrate, a command line Kernel profiling tool that we can download for free from MSFT. Kernrate is a general-purpose profiling tool for tracking CPU utilization by kernel-mode and user-mode processes.

• You can download KrView, the Kernrate Viewer from the following address: http://www.microsoft.com/whdc/system/sysperf/krview.mspx 
• Install it on your machine
• Navigate to the Kernrates folder inside the directory were you installed KrView (Default: C:\Program Files\KrView\Kernrates) and execute the binary for your OS. - For Vista and 2003 you can run Kernrate_i386_XP.exe.
• Let the profiler run when your system is on heavy load of CPU and then hit Ctrl-C

• Check the results for hits in the Kernel and % Total CPU time

In order to get a deeper understanding of the issue than just module reference, you'll need to zoom in. That is, for a particular module, like for instance Srv, Kernrate will track instruction pointer locations within the module to individual functions. This kind of granularity is often what you'll need to track down performance issues.

Because Kernate is tracking down function information you'll need to configure it to download symbol information as well, to do it follow the bellow steps:

• Grab the latest versions of dbghelp.dll and symsrv.dll from the Windows Debugging Tools and copy them in the directory where kernrate is installed.  
• Use the full download symbol path "-j symsrv*symsrv.dll**http://msdl.microsoft.com/download/symbols" with the command:
  Kernrate_i386_XP.exe -z Srv -j symsrv*symsrv.dll**http://msdl.microsoft.com/download/symbols
  the -z Srv will zoom in the Srv module Or
• Set _NT_SYMBOL_PATH=symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols - This will ensure you don't need to enter the following command everytime

The above is the result of the following command:

Kernrate_i386_XP.exe -z Srv -j symsrv*symsrv.dll**http://msdl.microsoft.com/download/symbols

I've ran into a post on the Sysinternals forum from a moderator on the forum that showed a script to automate the process check it out:

http://forum.sysinternals.com/forum_posts.asp?TID=11347

Grab the Debugging Tools for Windows (DTW) and toss a copy of symsrv.dll from [c:\Program Files\Debugging Tools for Windows] into [C:\Program Files\KrView\Kernrates].
 
Then, use the followning batch file to run Kernrate.  The batch file will set _NT_SYMBOL_PATH, go to the kernrates folder, check for symsrv.dll and if it is there it will invoke kernrate with some params, and then open the kernrates folder in Explorer.  Kernrate will collect data for 15 seconds and when it is done, there will be 2 output files.  output.kv can be loaded into KrView.xls, and stdouterr is kind of the "report" from the invocation of the batch file.  If you installed anything to a location other than the default, adjust paths in the batch file accordingly.  Also, you may want to further refine the length of time Kernrate runs (-s nnn) based off of how long it takes you to run the experiment.

@echo off

cd /d C:\Program Files\KrView\Kernrates
 
echo Before set: >stdouterr
set _NT_SYMBOL_PATH >>stdouterr
set_NT_SYMBOL_PATH=symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
echo After set: >>stdouterr
set _NT_SYMBOL_PATH >>stdouterr
 
if NOT exist symsrv.dll (
ECHO NO SYMSRV.DLL FOUND!
ECHO NO SYMSRV.DLL FOUND! >>stdouterr
) ELSE (
Echo Executing Kernrate_i386_XP.exe -v 0x7f -b 4 -f -ts -yr output.kv -s 15 ::
 
Kernrate_i386_XP.exe -v 0x7f -b 4 -f -ts -yr output.kv -s 15 1>>stdouterr 2>>&1
start "" "C:\Program Files\KrView\Kernrates"
)
 
pause

• Hmm mfehidk Module caught the eye, perhaps its time to contact Mcafee Support for help or an upgrade. Try to unregister and monitor for improvements in performance. Regsvr32 -u [Path to module]
• Press the Module button in Process Explorer for more info on binary version.

Monitor File System Activity on behalf of the System Process

Another way that can help on pointing what is going on on behalf of the System process, and since we can point Srv.sys in the previous case, is to monitor file system activity. For that purpose we can use process monitor and filter by Process Name System.

As you can imagine the troubleshooting targeted servers were DC's, as they were serving logon scripts at the time of the Filesystem capture.

HMM OPLOCK NOT GRANTED!

Solution

The process Monitor above gave us the clue to solve our problem. The issue concerned a bad designed logon script. The logon Script had a reference to a missing Network share. We remarked the entries to the missing share in the script,saved it and the CPU lowered from 60% to 5% in a second

This post had the purpose of helping everyone to troubleshoot System process issues and get everyone comfortable with the tools of the trade.

I will add some documentation as well as Download pages for the tools to aid and ease everyone's searches:

Downloads and Documentation:

Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Process Monitor: http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

KrView Kernrate Viewer: http://www.microsoft.com/whdc/system/sysperf/krview.mspx

Marks Blog The Case of the System Process CPU Spikes http://blogs.technet.com/markrussinovich/archive/2008/04/07/3031251.aspx

Ricardo Vicente

I ran accross an article on the Internet that helps customers to prepare their VHD's to deliver to PSS for troubleshooting support cases. That reminds me of the good old days.

• Clear out the machine of filler
  I recommend installing CCLEANER.EXE from http://ccleaner.com which deletes all temporary files from the Windows installation, Internet Explorer cache, the Recycle Bin, and any number of other "garbage"/"temp" directories in the VPC.
• Disable hibernation
  If you have hiberation turned on, there's a colossal sized file called hiberfil.sys in the root of the .VHD.  Go into Power options in Control Panel and disable hiberation to get rid of this file.
• Defragment the Virtual Hard Drive
  • Click Start, and then click My Computer.
  • Right-click the hard disk that you want to defragment, and then click Properties.
  • Click the Tools tab, and then click Defragment Now.
  • Click Defragment
  • You can also defragment the hard disk at the command prompt. To do this, follow these steps:
    Click Start, click Run, type cmd, and then click OK.
    Type defrag /? to view a list of the available options for the Disk Defragmenter tool.
• Capture the Virtual Disk Precompactor .ISO
  In the menu of Virtual PC windows for your machine, capture the Virtual Disk Precompactor .ISO file which is located at:
  c:\program files\microsoft virtual pc\virtual machine additions\virtual disk precompactor.iso
• Run the Precompactor
  Go to My Computer and double click on the mounted disk.  It will begin to run the precompactor software which will essentially zero out all the free space on the disk, making it available for elimination from the file by the Virtual Disk Wizard, which we will use later.  This process may take upwards of 15 minutes
• Shutdown the Virtual PC
• Run the VPC2007 Virtual Disk Wizard
  From the Virtual PC console, click File-Virtual Disk Wizard and "Edit an existing virtual disk".  It will ask you to identify the .VHD file to reduce in size - select it.
• Compact the .VHD
  Select the "Compact" option then select "Replacing the original file".  This will take a few minutes.
  

Hope this procedure helps most of you as much as it worked a million wonders for me,

Hi everyone, I wonder how many of you are pissed with the annoyance of having to refer to the systray whenever you have to unplug a Removable device in Windows XP. I found a way to create a shell extension for this that I'm going to share with you.

What about creating a shell extension that you could bring up by a right-click of a mouse.

Finding out the command line to call Safely Remove Hardware window and registry keys to create our shell extension:

• First I needed to know the command line that brings the Safely Remove Hardware window. for that I used Process Explorer from Sysinternals.

• I Pulled the Safely Remove Hardware window and used the Find Windows Process button to locate the process that loads it.

NOTE! From the picture above we found that our Safely Remove Hardware windows spawns from a rundll32.exe process, from the command line "C:\WINDOWS\system32\rundll32.exe" shell32.dll,Control_RunDLL hotplug.dll to be more precise.

• From this point I needed to locate the registry keys that bring the options to the context menu in windows explorer, and for that i used Process Monitor from Sysinternals.
• I pulled up the context menu while sniffing the file system with the Process monitor tool and...

NOTE! From the picture above we know what registry keys to touch to create our extension.

Creating Safely Remove Hardware shell extension

• Open regedit and go to the following registry key HKEY_CLASSES_ROOT\Drive
• Create a Key named Safely Remove Hardware
• Inside HKEY_CLASSES_ROOT\Drive\shell\Safely Remove Hardware create a Key named command
• Inside HKEY_CLASSES_ROOT\Drive\shell\Safely Remove Hardware\command edit the Default REG_SZ String with the Value Data: "C:\WINDOWS\system32\rundll32.exe" shell32.dll,Control_RunDLL hotplug.dll

NOW YOU'RE DONE NJOY

Hi there. It has been awhile since I've updated my blog so....

What should I bring up front on my next post? - I asked myself many times on the last couple of days. Besides, the blog is my own identity. I would feel a lesser engineer if I left my posts behind.

For the last couple of months I've been reviewing a popular 80's broadcast named Computer Chronicles.

COMPUTER CHRONICLES EPISODES

http://www.archive.org/details/computerchronicles

Hosted by Stewart Cheifet, Computer Chronicles was the world's most popular television program on personal technology during the height of the personal computer revolution. It was broadcast for twenty years from 1983 - 2002. The program was seen on more than 300 television stations in the United States and in over 100 countries worldwide, with translations into French, Spanish, Chinese, and Arabic. The series had a weekly television broadcast audience of over two million viewers. The series has been recognized for its journalistic excellence, winning a variety of journalistic awards including more than a dozen from the prestigious Computer Press Association. The series covered high-tech subjects around the world, having shot programs in such various locations as Czech Republic, France, Germany, Hong Kong, Hungary, India, Israel, Italy, Japan, Scotland, Spain, and Taiwan. Computer Chronicles was based in the Silicon Valley area of California. Many of the series programs are distributed on video to corporations and educational institutions for use in computer training. Computer Chronicles program segments have also been bundled with various computer text books by major publishers.

Gary Kildall was co-host of these series along with Stewart Cheifet in the picture bellow. Gary was one of the most important figures in the history of the personal computer science. I think it would be nice to review an episode as a tribute to this great figure.

A profile on computer pioneer Gary Kildall and the important contributions he made to the PC industry including the true story on how IBM ended up using MS-DOS rather than CP/M. Kildall developed CP/M, the first personal computer operating system. He was also a co-host on the early Computer Chronicles series. Includes comments by Gordon Eubanks, Symantec; Tom Rolander, DRI; Tim Bajarin, Creative Strategies; Lee Lorenzen, DRI; Jacqui Morby, TA Associates; Alan Cooper, CP/M applications developer. Originally broadcast in 1995. GARY KILDALL SPECIAL Download this episode in the link bellow: http://www.archive.org/details/GaryKild

SOME OF THE MOST DOWNLOADED VIDEOS INCLUDE:

CD ROM

Multimedia was the big deal in the early 90's as software publishers rushed to use the new storage medium for bigger and more interactive programs. This show looks at several new software titles on CD-ROM including Sherlock Holmes from ICOM Simulations, Jazz: A Multimedia History from Compton's New Media, and Battle Chess from MacPlay. Also a look at CD-ROM hardware including the TurboDuo from Turbo Technologies and the Portable CD-ROM Player from Sony and the Desktop Library CD-ROM system. Also featured is a visit to PC Week Labs for a review of new CD-ROM drives. Guests include Gina Smith of PC Computing and Jim Louderback of PC Week. Originally broadcast in 1993.             Download Link http://www.archive.org/details/CDROM10.3

COMPUTER GAMES

A look at the hottest computer games in the early 1990's. Included are Wordtris, Crisis in the Kremlin, Earl Weaver Baseball II, Railroad Tycoon, Playmaker Football 2.0, King's Quest V, The Games: Winter Challenge, and TSN (The Sierra Network). Also features a visit to Lucasfilm Games and their development process for titles such as Indiana Jones, The Fate of Atlantis, Monkey Island 2, and Lechuck's Revenge. Another segment looks at the InGolf Simulator from TOP Pro Sports, a game which actually teaches you how to play better golf. Guests include Gilman Louie, President of Spectrum Holobyte, Michael Brook of Electronic Arts, and Bruce Shelley of MicroProse. Originally broadcast in 1991.                               Download Link http://www.archive.org/details/episode_932

WINDOWS 95

It was the biggest ever launch of a software product with stores opening at midnight and buyers around the world waiting to get their hands on... an operating system! It was Windows 95. This show looks at the hoopla surrounding the launch of Windows 95, the product itself, and several Windows 95 add-on products including Microsoft Office, Viking RAM Chips, Uninstaller, Trispectives Professional, and The Hive. Originally broadcast in 1995. Download Link http://www.archive.org/details/CC1301_windows_95

MORE LINKS:

Fan SiTes:

http://www.cchronicles.com/

Gary Kildall's Bio:

http://en.wikipedia.org/wiki/Gary_Kildall