Bug Check 0x124 when Sysprep generalizes Windows 10

We have recently been preparing for the deployment of a big number of new desktop computers. The plan was to follow the Microsoft ways using WDS and/or MDT, whose procedures generally include setting up a referencing computer and customizing it, then capturing the image to deploy.

The process went along very well during our early stage tests on virtual machines, until two samples of the physical machines were delivered to us for further preparation, when things began to get complicated.

The model was DELL Optiplex 3240 AIO, an all-in-one computer, and the problem was most of the time when Sysprep was run to generalize the OS, it turned out a blue screen of death, with Bug Check error type WHEA_UNCORRECTABLE_ERROR, no matter if firmware was set in UEFI mode or legacy BIOS mode.

Performing postmortem analyses on the Minidump files, I found out the error code was 0x124, and the first parameter was always 0x4, which means an uncorrectable PCI Express error occurred. After running !analyze -v, WinDbg reported the problem was probably caused by GenuineIntel, but I highly doubted it was the CPU to blame.

Most of the documents, including MSDN and the Microsoft community Wiki, says that this Bug Check is almost always caused by physical hardware failures, especially overclocking. In spite of that, I still could not convince myself. Obviously, all of the two sample machines were showing the same symptom, and if it were a hardware issue, the quality would be disastrous. After all, it’s DELL.

Finally, I googled ‘sysprep bsod 124’ and found one case on Microsoft TechNet forum, titled ‘BSOD when sysprep Windows 10‘, which was nearly identical to our situation: all-in-one computer, same Bug Check error, same type number, only the brand is HP. In the replies, someone provided another kernel debugger helper command, !errrec, which shows detailed WHEA error records given the address, i.e. the second parameter in the Bug Check.

Later, I found the command was already provided in the MSDN document of Bug Check 0x124. The solution could come much earlier, had I been patient enough to follow the document!

In the output of !errrec the related Device Ids were revealed.

  • 10ec:8168, Realtek Gigabit Ethernet
  • 8086:24f3, Intel Dual Band Wireless-AC 8260

All of them are network adapters.

Since it’s not practical to remove any hardware from inside of an all-in-one computer, we decided to firstly try uninstalling all network adapters from Device Manager before doing Sysprep, and it worked!

When the network devices are removed manually from Device Manager one by one, with Sysprep run immediately afterwards, blue screens are never observed again. The method really worked throughout our subsequent generalization and capturing processes.

But why does such a serious problem happen in such a fundamental procedure? Obviously, the scenario has never been tested by the manufacturers, neither DELL nor Realtek, Intel, etc. Don’t DELL’s teams use Sysprep any more? Guess so and not.

  • Sysprep may be still in use. When doing the search, I saw some comments on Sysprep, which recommend building a reference computer offline, to prevent background tasks, esp. Windows Updates from altering the system without solicitation. Maybe DELL make their images in a closed lab environment where our scenario never happens.
  • Sysprep may be fading out. With prevailing DISM techniques, servicing an image offline might be no longer a question but perhaps a best practice. When altering an image offline, device drivers never sneak in so our scenario will never happen.
Advertisements

Use Write-Verbose to output messages when writing PowerShell scripts

It seems to be intuitive to use Write-Host to output debugging or informative messages to the console when writing and debugging PowerShell scripts, as we usually do in the “hello, world” examples, but it’s not right.

The correct way of doing this looks like the following function:

function my-function
{
    [CmdLetBinding()]
    param()

    Write-Verbose "the message"
}

By default, the verbose messages won’t display. To show them, call the function with the standard Verbose option:

my-function -Verbose

ref.

What does ‘Exp’ mean in source code files from a CVS repository?

We often see an ‘Exp’ in source code files from a CVS repository, eg.

$Id: samp.c,v 1.5 1993/10/19 14:57:32 ceder Exp $

$Header: /projects/compbio/cvsroot/CompbioCVSDoc/cvs-manual/cvs_76.html,v 1.1 1997/04/19 23:10:26 markd Exp $

What does it mean?

Answer from Guide to CVS commands:

Any identifier is acceptable for state. A useful set of states is `Exp’ (for experimental), `Stab’ (for stable), and `Rel’ (for released). By default, the state of a new revision is set to `Exp’ when it is created. The state is visible in the output from cvs log, and in the $Log$ and $State$ keywords. Note that CVS uses the dead state for its own purposes; to take a file to or from the dead state use commands like cvs remove and cvs add, not cvs admin -s.

Overlay Images with EXIF Timestamp using ImageMagick

Sometimes we may need to annotate an image with its timestamp. Like the following:

DSC02268 => DSC02268

Here’s how to do this:

on Windows 7, open cmd with delayed expansion on.

cmd /v:on

set PATH to ImageMagick.

path C:\Program Files (x86)\ImageMagick-6.3.3-Q16;%path%

change current directory to where image files exist.

cd /d D:\TEMP\2014-10-22

run:

if not exist withdate mkdir withdate
for %i in (*.jpg) do @(
  set o=withdate\%i
  echo %i =^> !o!
  for /f "tokens=1-2" %x in ('convert -ping -format "%[EXIF:DateTimeOriginal]" %i info:') do @(
    set d=%x
    set d=!d::=-! %y
  )
  for /f %x in ('convert -ping -format "%[height]" %i info:') do @(
    set /a fontsize=%x/25
    set /a padding=%x/50
    set /a strokewidth=%x/25/12
  ) >nul
  convert -gravity SouthEast -font Tahoma -pointsize !fontsize! ^
    -stroke black -strokewidth !strokewidth! -annotate +!padding!+!padding! "!d!" ^
    -stroke none -fill white -annotate +!padding!+!padding! "!d!" ^
    %i !o!
)

ref.

zfs on FreeBSD 10 snippets

1. easily identify new disks with gpt label

1.1 gpart create -s gpt da0
1.2 gpart add -t freebsd-zfs -a 100M -s 136G -l nas01.0.bay5 da0
||
gpart add -t freebsd-zfs -a 100M -s 136G da1
gpart modify -i 1 -l nas01.ext1.bay1 da15
1.3 true > /dev/da0p1 # workaround for kern/154226
1.4 gpart show -lp

2. create/import zpool (transactional object layer; pooled storage layer)

zpool create tank mirror gpt/nas01.0.bay1 gpt/nas01.0.bay5
zpool attach tank mirror-0 gpt/nas01.0.bay3
zpool add tank mirror gpt/nas01.0.bay2 gpt/nas01.0.bay6
zpool add tank spare gpt/nas01.0.bay8
zpool import

3. create zfs dataset (zfs posix layer; zfs volume emulator)

zfs create -o quota=5G -o copies=2 tank/home
zfs set compression=on tank/home
zfs create -o atime=off -o exec=off -o setuid=off -o logbias=throughput tank/ghost

4. move var and tmp

# ref. freebsd on zfs
/etc/rc.conf: zfs_enable=”YES”

5. samba

# … pkg install samba42 …
# ref. /usr/local/etc/smb.conf
/etc/rc.conf: samba_enable=”YES”

6. nfs

/etc/rc.conf # ref. nfs on freebsd
zfs set sharenfs=’-network 192.168.117.0/24 -maproot=0′ tank/vmware

7. iscsi

# ref. iscsi on freebsd

8. tuning for vmware esxi

zfs set atime=off tank/vmware
# compromise data integrity for performance, ups required
zfs set sync=disabled tank/vmware

9. set up monitor

# /etc/periodic.conf:
daily_status_zfs_enable=”YES”
daily_scrub_zfs_enable=”YES”

A. view and monitor

zfs get all tank/home
zfs list -t all -o name,type,creation -s creation
zpool iostat -v tank 5

B. backup

B.0 preparation:

#tail -F /var/log/messages
#dd if=/dev/random bs=1m count=10240 of=/dev/ada8
gpart create -s gpt ada8
gpart add -t freebsd-zfs -a 4K -s 1953514544 -l nas01.bak01 ada8
geli init gpt/nas01.bak01

B.1 first-time:

geli attach gpt/nas01.bak01
zpool create -R /bak01root bak01 gpt/nas01.bak01.eli
zfs snapshot -r tank@2013-03-25
zfs hold -r latest-backup tank@2013-03-25
zfs send -R tank@2013-03-25 | zfs receive -duvF bak01
zpool export bak01
geli detach gpt/nas01.bak01

B.2 incremental:

zfs snapshot -r tank@`date +%F`
zfs hold -r latest-backup tank@`date +%F`
geli attach gpt/nas01.bak01
zpool import -N -R /bak01root bak01
zfs list -H -d 1 -t snapshot -o name tank | xargs zfs holds
#zfs list -H -t snapshot -o name | grep ‘^bak01.*@yyyy-mm-dd’ | xargs -n1 zfs rollback
zfs send -R -i yyyy-mm-dd tank@`date +%F` | zfs receive -duvF bak01
zpool export bak01
geli detach gpt/nas01.bak01
# perform (B.3), and then
zfs list -H -d 1 -t snapshot -o name tank | xargs zfs holds
zfs release -r latest-backup tank@yyyy-mm-dd
#zfs get userrefs | sort -k 3

B.3 verify on another machine booted from LiveCD:

geli attach gpt/nas01.bak03
zpool import -N bak03
zpool scrub bak03
gstat -p
zpool status
zpool export bak03
geli detach gpt/nas01.bak03

C. replace disk

gpart show -lp da0
zpool replace tank gpt/nas01.0.bay1 gpt/nas01.0.bay8
||
zpool replace tank 3545861042431994935 gpt/nas01.0.bay5
zpool detach tank gpt/nas01.0.bay1
gpart modify -i 1 -l nas01.0.bay1.FAIL-`date +%Y%m%d` da0
# replace hardware
=> (1) partition and label the disk immediately after insertion
=> (F) erase zfs metadata if “zpool replace” complains
zpool add tank spare gpt/nas01.0.bay1

D. reduce mirror level

zpool detach tank gpt/nas01.0.bay3

E. destroy dataset

zfs destroy tank/test

F. erase zfs vdev metadata (tcsh arithmetic)

# cf. zpool labelclear
set p=gpt/nas01.0.bay5
zdb -l /dev/${p}
set s=`diskinfo ${p} | cut -f 3`
@ s = $s / 1024 / 512 – 1
dd if=/dev/zero of=/dev/${p} bs=512K count=1
dd if=/dev/zero of=/dev/${p} bs=512K count=2 seek=$s
# ref. http://www.slideshare.net/relling/zfs-tutorial-usenix-lisa09-conference p.22

NetFlow vs sFlow

Nutshell: NetFlow for routers and firewalls, while sFlow for Ethernet switches.

NetFlow v9 ==> IPFIX (IETF) draft

NetFlow: High CPU usage. Some vendor with hardware.
sFlow: Free license for vendors. Hardware (expansion card) is the only choice.

NetFlow records every packet, header and payload truncated at 1200 bytes. Flexible, can get URLs, hostnames, etc. Sampled NetFlow exists.
sFlow takes samples. Designed for Ethernet.

ref.

  1. NetFlow vs. sFlow for Network Monitoring and Security: The Final Say
  2. NetFlow or sFlow: which is the open standard?

Differences between partners and resellers

Original from: http://www.cloudcentrics.com/?p=827

  • Partners understand solutions can be complicated.
  • Partners usually have Solution Architects that work with engineering and project management teams to make sure all parties understand all aspects of a solution.
  • Designs are created, reviewed, and explained to customers before installation begins.
  • Partners and customers are able install and test solutions together before engineers come to your location and start unboxing equipment and plugging in equipment.
  • Partners have true integration facilities that can be used to configure your entire solution.
  • Partners invest in day-two support options for their clients. They provide options for onsite engineering and/or call center support.

VMware vShield Endpoint may consume too much memory on Windows XP

The problem affects:

VMware vCenter Server 5.0 Update 1 and modules 5.0.0U1
(VMware-VIMSetup-all-5.0.0-639890.iso)
    VMware Tools 8.6.5 build-621624
      vsepflt.sys 5.0.0.0 build-443031

The problem is fixed in:

VMware Tools 8.6.5 build-731933
    vsepflt.sys 5.0.0.1 build-652273

VMware vShield Endpoint Thin Agent 1.0 Update 3 Release Notes

After installation of Thin Agent drivers (vShield Endpoint 1.0.0, vShield Endpoint 1.0.0 Update 1, or vShield Endpoint 1.0.0 Update 2) on Windows XP/2003 guests, overall memory usage of the guest VM may increase substantially with all processes consuming higher amounts of memory than usual. This behavior is not observed in later versions of Windows viz. Vista and later. It results from a combination of the way the Thin Agent reads a file during its scan and Windows internal behavior.

Computer certificate may fail VMware View Connection Server

As of VMware View Connection Server 5.1.0.

In an Active Directory environment, using the computer certificate signed by Enterprise CA may cause VMware View Connection Server fail to start. To solve the problem: 1) use 2003 template, 2) make private key exportable.

Detail:

At CA,
1. duplicate the template with 2003 compatibility, private key exportable
2. make the template available to the CA

At server,
1. enroll again for a certificate
2. set friendly name to “vdm”
3. restart the VMware View Connection Server service

cf. http://derek858.blogspot.com/2012/05/vmware-view-51-installation-part-1-view.html