I’ve been exploring the Sync Framework for use in a couple of projects I have going and PowerShell is my preferred exploratory environment.

It was a bit of fun, since I got to work with eventing for the first time in V2.

First, I downloaded the Sync Framework Software Development Kit.  That provided me with the Sync Framework runtime as well as some documentation.

The easiest way for me to get started was to take one of the samples and convert that to PowerShell.

I’m going to walk along the MSDN Sample and provide the equivalent PowerShell, as well as any changes I made to make it feel more PowerShell-y.

Setting Synchronization Options

We are working with the File Sync Provider First up is setting the FileSyncOptions.  FileSyncOptions are an enumeration (a limited list defined in code that maps to certain values) whose values are controlled by setting the appropriate bits to indicate the presence or absence of a flag.  Mark Schill has a great post about how to set bitwise operations.

$options = [Microsoft.Synchronization.Files.FileSyncOptions]::ExplicitDetectChanges
$options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecycleDeletedFiles
$options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecyclePreviousFileOnUpdates
$options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecycleConflictLoserFiles

Specifying a Static Filter

With the File System provider, we can provide filters to include or exclude files and directories.

$FileNameFilter and $SubdirectoryNameFilter are parameters that take strings or string arrays.

$filter = New-Object Microsoft.Synchronization.Files.FileSyncScopeFilter
if ($FileNameFilter.count -gt 0)
{
   $FileNameFilter | ForEach-Object { $filter.FileNameExcludes.Add($_) }
}
if ($SubdirectoryNameFilter.count -gt 0)
{
   $SubdirectoryNameFilter | ForEach-Object { $filter.SubdirectoryExcludes.Add($_) }
}

Performing Change Detection

After configuring the filter, we examine the folders and files located at the paths specified.  If there has not been any previous synchronization, a metadata file will be created in each location to track any changes, updates, and deletes for later synchronization.

function Get-FileSystemChange()
{
    param ($path, $filter, $options)
    try
    {
        $provider = new-object Microsoft.Synchronization.Files.FileSyncProvider -ArgumentList $path, $filter, $options
        $provider.DetectChanges()
    }
    finally
    {
        if ($provider -ne $null)
        {
            $provider.Dispose()
        }
    }
}

Get-FileSystemChange $SourcePath $filter $options
Get-FileSystemChange $DestinationPath $filter $options

Handling Conflicts

Conflict resolution in the Sync Framework happens at the at the event level.  An event is merely something that happens that can trigger other actions.  Using Register-ObjectEvent, we can associate one or more scriptblocks with an event.

First, I defined scriptblocks to handle the conflicts.  There is an enumeration, the ConflictResolutionAction enumeration, that provides some options for dealing with conflicts.  For this example, we are going to pick the source object as the winner for any conflicts.

You will also notice another type of conflict defined, and that is a Constraint conflict.  That can occur when an object of the same name is added on both sides in between synchronizations.  The resolution options for these conflicts can be found in the ConstraintConflictResolutionAction enumeration.

$ItemConflictAction =   {
    $event.SourceEventArgs.SetResolutionAction([Microsoft.Synchronization.ConflictResolutionAction]::SourceWins)
    [string[]]$global:FileSyncReport.Conflicted += $event.SourceEventArgs.DestinationChange.ItemId
}
$ItemConstraintAction = {
     $event.SourceEventArgs.SetResolutionAction([Microsoft.Synchronization.ConstraintConflictResolutionAction]::SourceWins)
     [string[]]$global:FileSyncReport.Constrained += $event.SourceEventArgs.DestinationChange.ItemId
}            

# Configure the events for conflicts or constraints for the source and destination providers
$destinationCallbacks = $destinationProvider.DestinationCallbacks
Register-ObjectEvent -InputObject $destinationCallbacks -EventName ItemConflicting -Action $ItemConflictAction | Out-Null
Register-ObjectEvent -InputObject $destinationCallbacks -EventName ItemConstraint -Action $ItemConstraintAction | Out-Null            

$sourceCallbacks = $SourceProvider.DestinationCallbacks
Register-ObjectEvent -InputObject $sourceCallbacks -EventName ItemConflicting -Action $ItemConflictAction | Out-Null
Register-ObjectEvent -InputObject $sourceCallbacks -EventName ItemConstraint -Action $ItemConstraintAction | Out-Null

We also see for the first time in the script blocks a variable called $event.  This is an automatic variable exposed by the event and provides us information that we can use in our action.

Finally, I’m updating a variable in the global scope.  There probably is a better way to handle this, but scriptblocks executed in response to events only have access to the global scope and any of the automatic variable exposed to it.  Therefore, I use a variable in the global scope to gather my reporting information.

Synchronizing Two Replicas

To start to synchronize the two sides, first we set up the synchronization via a SyncOrchestrator and assign it the local and remote providers, as well as defining the direction of the synchronization.  In this example (sticking with the format from MSDN, we will do an Upload, which is in the SyncDirectionOrder enumeration (other options are Download, DownloadAndUpload, and UploadAndDownload). 

# Create the agent that will perform the file sync
$agent = New-Object  Microsoft.Synchronization.SyncOrchestrator
$agent.LocalProvider = $sourceProvider
$agent.RemoteProvider = $destinationProvider            

# Upload changes from the source to the destination.
$agent.Direction = [Microsoft.Synchronization.SyncDirectionOrder]::Upload            

Write-Host "Synchronizing changes from $($sourceProvider.RootDirectoryPath) to replica: $($destinationProvider.RootDirectoryPath)"
$agent.Synchronize();

To achieve two way synchronization, we will do the upload twice, reversing the order of the providers.

Invoke-OneWayFileSync -SourcePath $SourcePath -DestinationPath $DestinationPath -Filter $null -Options $options
Invoke-OneWayFileSync -SourcePath $DestinationPath -DestinationPath $SourcePath -Filter $null -Options $options

Full Example

I modified the example to write out a custom object (and the logging is in the variable in the global scope as noted in the Handling Conflicts section) with the results of the synchronization (rather than logging it to the console).

In all, my translation is pretty similar to the example code, but there are some differences. 

# Requires -Version 2
# Also depends on having the Microsoft Sync Framework 2.0 SDK or Runtime
# --SDK--
# http://www.microsoft.com/downloads/details.aspx?FamilyID=89adbb1e-53ff-41b5-ba17-8e43a2e66254&displaylang=en
# --Runtime--
# http://www.microsoft.com/downloads/details.aspx?FamilyId=109DB36E-CDD0-4514-9FB5-B77D9CEA37F6&displaylang=en
#
#            

[CmdletBinding(SupportsShouldProcess=$true)]
param (
    [Parameter(Position=1, Mandatory=$true, ValueFromPipelineByPropertyName=$true)]
    [Alias('FullName', 'Path')]
    [string]$SourcePath
    , [Parameter(Position=2, Mandatory=$true)]
    [string]$DestinationPath
    , [Parameter(Position=3)]
    [string[]]$FileNameFilter
    , [Parameter(Position=4)]
    [string[]]$SubdirectoryNameFilter
)
<#
    .Synopsis
        Synchronizes to directory trees
    .Description
        Examines two directory structures (SourcePath and DestinationPath) and uses the Microsoft Sync Framework
        File System Provider to synchronize them.
    .Example
        An example of using the command
#>
begin
{
    [reflection.assembly]::LoadWithPartialName('Microsoft.Synchronization') | Out-Null
    [reflection.assembly]::LoadWithPartialName('Microsoft.Synchronization.Files') | Out-Null            

    function Get-FileSystemChange()
    {
        param ($path, $filter, $options)
        try
        {
            $provider = new-object Microsoft.Synchronization.Files.FileSyncProvider -ArgumentList $path, $filter, $options
            $provider.DetectChanges()
        }
        finally
        {
            if ($provider -ne $null)
            {
                $provider.Dispose()
            }
        }
    }            

    function Invoke-OneWayFileSync()
    {
        param ($SourcePath, $DestinationPath,
                    $Filter, $Options)
        $ApplyChangeJobs = @()
        $AppliedChangeJobs = @()
        try
        {
            # Scriptblocks to handle the events raised during synchronization
            $AppliedChangeAction = {
                $argument = $event.SourceEventArgs
                switch ($argument.ChangeType)
                {
                    { $argument.ChangeType -eq [Microsoft.Synchronization.Files.ChangeType]::Create } {[string[]]$global:FileSyncReport.Created += $argument.NewFilePath}
                    { $argument.ChangeType -eq [Microsoft.Synchronization.Files.ChangeType]::Delete } {[string[]]$global:FileSyncReport.Deleted += $argument.OldFilePath}
                    { $argument.ChangeType -eq [Microsoft.Synchronization.Files.ChangeType]::Update } {[string[]]$global:FileSyncReport.Updated += $argument.OldFilePath}
                    { $argument.ChangeType -eq [Microsoft.Synchronization.Files.ChangeType]::Rename } {[string[]]$global:FileSyncReport.Renamed += $argument.OldFilePath}
                }
            }            

            $SkippedChangeAction = {
                [string[]]$global:FileSyncReport.Skipped += $event.SourceEventArgs.CurrentFilePath            

                if ($event.SourceEventArgs.Exception -ne $null)
                {
                    Write-Error '[' + "$($event.SourceEventArgs.Exception.Message)" +']'
                }
            }            

            # Create source provider and register change events for it            

            $sourceProvider = New-Object Microsoft.Synchronization.Files.FileSyncProvider -ArgumentList $SourcePath, $filter, $options
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $SourceProvider -EventName AppliedChange -Action $AppliedChangeAction
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $SourceProvider -EventName SkippedChange -Action $SkippedChangeAction             

            $ApplyChangeJobs += $SourceApplyChangeJob            

            # Create destination provider and register change events for it
            $destinationProvider = New-Object Microsoft.Synchronization.Files.FileSyncProvider -ArgumentList $DestinationPath, $filter, $options
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $destinationProvider -EventName AppliedChange -Action $AppliedChangeAction
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $destinationProvider -EventName SkippedChange -Action $SkippedChangeAction            

            $ApplyChangeJobs += $DestApplyChangeJob            

            # Use scriptblocks for the SyncCallbacks for conflicting items.
            $ItemConflictAction =   {
                $event.SourceEventArgs.SetResolutionAction([Microsoft.Synchronization.ConflictResolutionAction]::SourceWins)
                [string[]]$global:FileSyncReport.Conflicted += $event.SourceEventArgs.DestinationChange.ItemId
            }
            $ItemConstraintAction = {
                $event.SourceEventArgs.SetResolutionAction([Microsoft.Synchronization.ConstraintConflictResolutionAction]::SourceWins)
                [string[]]$global:FileSyncReport.Constrained += $event.SourceEventArgs.DestinationChange.ItemId
            }            

            #Configure the events for conflicts or constraints for the source and destination providers
            $destinationCallbacks = $destinationProvider.DestinationCallbacks
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $destinationCallbacks -EventName ItemConflicting -Action $ItemConflictAction
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $destinationCallbacks -EventName ItemConstraint -Action $ItemConstraintAction             

            $sourceCallbacks = $SourceProvider.DestinationCallbacks
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $sourceCallbacks -EventName ItemConflicting -Action $ItemConflictAction
            $AppliedChangeJobs += Register-ObjectEvent -InputObject $sourceCallbacks -EventName ItemConstraint -Action $ItemConstraintAction             

            # Create the agent that will perform the file sync
            $agent = New-Object  Microsoft.Synchronization.SyncOrchestrator
            $agent.LocalProvider = $sourceProvider
            $agent.RemoteProvider = $destinationProvider            

            # Upload changes from the source to the destination.
            $agent.Direction = [Microsoft.Synchronization.SyncDirectionOrder]::Upload            

            Write-Host "Synchronizing changes from $($sourceProvider.RootDirectoryPath) to replica: $($destinationProvider.RootDirectoryPath)"
            $agent.Synchronize();
        }
        finally
        {
            # Release resources.
            if ($sourceProvider -ne $null) {$sourceProvider.Dispose()}
            if ($destinationProvider -ne $null) {$destinationProvider.Dispose()}
        }
    }            

    # Set options for the synchronization session. In this case, options specify
    # that the application will explicitly call FileSyncProvider.DetectChanges, and
    # that items should be moved to the Recycle Bin instead of being permanently deleted.            

    $options = [Microsoft.Synchronization.Files.FileSyncOptions]::ExplicitDetectChanges
    $options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecycleDeletedFiles
    $options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecyclePreviousFileOnUpdates
    $options = $options -bor [Microsoft.Synchronization.Files.FileSyncOptions]::RecycleConflictLoserFiles
}
process
{
    $filter = New-Object Microsoft.Synchronization.Files.FileSyncScopeFilter
    if ($FileNameFilter.count -gt 0)
    {
       $FileNameFilter | ForEach-Object { $filter.FileNameExcludes.Add($_) }
    }
    if ($SubdirectoryNameFilter.count -gt 0)
    {
       $SubdirectoryNameFilter | ForEach-Object { $filter.SubdirectoryExcludes.Add($_) }
    }            

    # Perform the detect changes operation on the two file locations
    Get-FileSystemChange $SourcePath $filter $options
    Get-FileSystemChange $DestinationPath $filter $options            

    # Reporting Object - using the global scope so that it can be updated by the event scriptblocks.
    $global:FileSyncReport = New-Object PSObject |
        Select-Object SourceStats, DestinationStats, Created, Deleted, Overwritten, Renamed, Skipped, Conflicted, Constrained            

    # We don't need to pass any filters here, since we are using the file detection that was previously completed.
    # this will only 
    $global:FileSyncReport.SourceStats = Invoke-OneWayFileSync -SourcePath $SourcePath -DestinationPath $DestinationPath -Filter $null -Options $options
    $global:FileSyncReport.DestinationStats = Invoke-OneWayFileSync -SourcePath $DestinationPath -DestinationPath $SourcePath -Filter $null -Options $options            

    # Write result to pipeline
    Write-Output $global:FileSyncReport
}

I’m updating Crystal Reports and trying to determine which reports might have been affected by some schema changes or functional changes in how the data was being stored. 

The problem I’ve had is that when there are a large number of reports, it is very time consuming to open each one, look at it, and see if it contains any affected tables or views.

I’ve had to deal with this in my previous role as well.  After feeling the pain a few times, I turned my intern loose on the problem and shelved the problem as “just another pain in dealing with Crystal Reports”.

Now, I’m back dealing with Crystal Reports more frequently and in the position to have to possibly update around 30 or 40 reports that were written before I started.

I’ve recently had a bit of exposure to the object model for the .NET API for Crystal Reports and thought maybe I could leverage that through PowerShell and whip together a quick script to help me list out the tables in each report.

It turned out to be painfully easy… 

[reflection.assembly]::LoadWithPartialName('CrystalDecisions.Shared')
[reflection.assembly]::LoadWithPartialName('CrystalDecisions.CrystalReports.Engine')
$report = New-Object CrystalDecisions.CrystalReports.Engine.ReportDocument
$report.load($pathToScript)
$report.Database.Tables | Select-Object -expand Name
$report.Dispose()

After I got the basics, I poked around and updated the script further (and posted it on PoshCode).

The full script also accesses the first level of subreports and retrieves their tables as well.

NOTE: Requires either the Crystal Report Runtime (Visual Studio 2008)  or Visual Studio to be installed.

A lot has happened since I last had the opportunity to post. 

Quick recap:

• PowerShell v2 is now out and available for XP, Server 2003, Vista, Server 2008, and installed and on by default for Windows 7 and Server 2008 R2
• SharePoint 2010 has an insane number (492)
• There is an open source cmdlet and help designer
• Try PowerShell is a great way to explore
• The PowershellPack provides some great modules that get you jump started with V2.
• Azure has cmdlets for management
• There is introductory PowerShell material in the Server 2008 R2 Developer Training Kit
• There are some PowerShell for Developers training videos on Channel 9
• James Kovacs released psake V2.00
• PsTFS has hit V1 – which is great for me now..
• I’m sure I’ve missed a ton of other cool PowerShell stuff that is happening too..

And that leads me to what I’m doing now. 

In August, I left my job as a IT Specialist for a municipal police department and took a job heading up research and development for ProPhoenix, which is a software development company that make software for public safety agencies (police, fire, corrections, municipal courts).

I now live in a more software development focused world and will get to spend more time focusing on using PowerShell to automate my current activities, explore new frameworks, and to build a management API for our applications.  I can’t wait to make PowerShell a requirement for our server products installation so I can get our support staff to start leveraging the capabilities PowerShell brings to the table.

I started looking a little deeper at error handling in PowerShell after this StackOverflow question.

PowerShell has two kinds of errors – terminating errors and non-terminating errors.

Terminating errors are the errors that can stop command execution cold.  Non-terminating errors provided an additional challenge, as you need to be notified of failed operations and continue with pipeline operations.  To deal with this issue and to provide additional output options, PowerShell employs the concept of streams.  There are three additional streams (other than the primary pipeline stream) available in PowerShell – Verbose, Warning, and Error (and .  We’ll be concerning ourselves with the output from the Error stream.

There are some automatic variables in the shell that deal with errors as well.  $ErrorActionPreference is a variable that describes how PowerShell will treat non-terminating errors.  $ErrorActionPreference has four options: “Continue” (the default), “SilentlyContinue”, “Inquire”, and “Stop”.  $error is an array of all the errors that have occurred in the current session up to the number specified in $MaximumErrorCount.  $? is a boolean value that is $true if the previous operation succeeded and $false if it did not.

PowerShell does have some built in flexibility to turn non-terminating errors into terminating errors.  Based on our setting of $ErrorActionPreference, PowerShell will treat a non-terminating error differently.  If $ErrorActionPreference is set to ‘”Stop”, PowerShell will treat the errors from that scope and all sub scopes (unless explicitly overridden) as terminating errors.  If “Inquire” is chosen as the $ErrorActionPreference, then PowerShell will prompt the user at the console upon each error to ask whether it should continue (treat as non-terminating) or halt (treat as terminating), as well as providing an option to drop into a nested prompt.

$ErrorActionPreference is a global preference that can be set, but the PowerShell runtime also allows more granular control by providing common parameters for -ErrorAction and -ErrorVariable (shorthand -EA and –EV) which allow you to determine per cmdlet (and in V2 per advanced function) how errors should be handled.  The –ErrorAction parameter takes the same options as $ErrorActionPreference.

So what happens when you encounter an error?

If you hit a terminating error, your script, function, or command will stop.  The error will be logged to the $error variable and written to the error stream.  The $? will be set to false.  You can use the trap construct (or the try/catch/finally construct in V2) to deal with terminating errors and we’ll cover that further in the fourth post in this series.

If you hit a non-terminating error, the result will depend on the $ErrorActionPreference in the scope or the –ErrorAction parameter.  If the $ErrorActionPreference or –ErrorAction parameter is set to ‘Continue’, the error will be written to the error stream, added to the $error array, and the $? will be set to $false.  ‘SilentlyContinue’ will not write an error to the Error stream, but $error and $? are updated.  Finally, ‘Inquire’ provides you an option at each error as to whether to treat it as terminating or non-terminating.  If treated as a non-terminating error, the error will be treated like the $ErrorActionPreference or –ErrorAction parameter is ‘Continue’.

Now, I’ve mentioned that some terminating and non-terminating errors both write to the Error stream, so where does that actually get reported back to the user?  If an error has been written to the error stream, it will be written to the output stream (which may surface differently based on the PowerShell host) separately from the pipeline output unless it is redirected.  This means that the exception objects, which are how the errors are represented, are not saved to a variable if you are assigning the pipeline output to a variable.

For example:

$results = Get-WMIObject Win32_Bios –ComputerName localhost, ComputerThatDoesNotExist
Get-WmiObject : The RPC server is unavailable. (Exception from HRESULT: 0×800706BA)
At line:1 char:22
+ $results = Get-WmiObject  <<<< Win32_Bios -ComputerName localhost, ComputerThatDoesNotExist

creates this output to the screen, but $results will only hold the results of the successful WMI query.  This is where the –ErrorVariable comes in.  You can specify a variable to hold the exceptions generated by a particular cmdlet (or in V2 advanced function).  Specifying a variable does not remove an item from being written to the error stream and displayed as output.

In a console session, you can redirect the error stream with the redirection operators ( 2> or 2>>) or you can merge the error stream with the output stream (2>&1) and write the output to a file, the pipeline, or assign it to a variable.

Up next:

• Error Objects
• Tracing Errors
• Trapping Errors
• Reporting Errors

Other great PowerShell Error Handling posts:

• http://huddledmasses.org/trap-exception-in-powershell/
• http://blogs.msdn.com/powershell/archive/2006/11/03/erroraction-and-errorvariable.aspx
• http://tfl09.blogspot.com/2009/01/error-handling-with-powershell.html
• http://weblogs.asp.net/adweigert/archive/2007/10/10/powershell-try-catch-finally-comes-to-life.aspx
• http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chapter-11-finding-and-avoiding-errors.aspx

The July CTP release of the Windows Azure SDK contains a new sample project called PowerShellRole which demonstrates that PowerShell is available in the cloud!

Previous versions of the CTP have come with a sample Provider which you could use to access Azure storage (blobs, queues, and tables), but this actually provides demonstration of creating runspaces and executing pipelines in the cloud.

Now to see what version is running in the cloud!

Garbage collection is a process that the .NET Framework (upon which the PowerShell runtime works) uses to manage memory.  The garbage collection (for applications – services are handled a bit differently) process basically covers X steps:

1. Identify objects that won’t be used
2. Delete them from memory
3. Compact the space to make room for new objects

The first step is the critical point for PowerShell users.  You may have noticed how the memory used in a PowerShell session can steadily climb (or suddenly climb if you’re dealing with a large number of objects).  I talked about that in this tip.  The .NET garbage collector looks for objects that are still in use or could possibly be in use in the near future (this is a gross oversimplification – if you are really interested in a detailed explanation of how garbage collection works in the .NET Framework, check out this article.)

What this means to scripters who are running into memory pressure issues is that the garbage collector will not reclaim any objects you have actively referenced (meaning that you’ve assigned them to a variable) in your current or higher scope. 

Since this can sound a bit convoluted let’s look at an example. 

I read a large file to parse it and leave the contents of the original stored in a variable

PS> $MyLargeFile =  Get-Content MyLargeFile.txt
PS> Foreach ($line in $MyLargeFile) {
DoSomething $line | Out-File NewUpdatedLargeFile.txt
}

One of the problems with the above pattern is that the original file remains saved in memory and doesn’t get released unless you explicitly remove the reference, either by using Remove-Variable or Remove-Item or by changing what $MyLargeFile points to by assigning another value to it. 

This isn’t VBScript where you have to set values to “Nothing”, but you should be aware of what larger object sets you are retaining in memory. 

I just finished listening to the latest Herding Code podcast (#52) where the hosts (K. Scott Allen, Kevin Dente, Scott Koon, and Jon Galloway) talked with Alan Stevens ( C# MVP and ASP Insider) and G. Andrew Duthie (author and Microsoft Developer Evangelist) about a debate that began on Twitter regarding “Real Software Development vs Microsoft Bubble Development”.

What does that have to do with PowerShell and administrative tools?  The specifics of their conversation don’t have a lot of relevance to administrators and scripters, but one of the directions that their conversation took really resonated with me.

Alan throws the first punch – He likes Herding Code because it’s about real software development rather than development in the Microsoft bubble.  It’s about the tool users rather than the tool builders and it’s about honest feedback.

As administrators, we need to make sure the developers of the applications that we use and administer provide us the tools we need to efficiently run our networks.  Microsoft has gotten the message loud and clear.  Windows 7, Server 2008 R2, and TechEd 2009 LA confirmed that.  There weren’t many sessions where you didn’t hear something about PowerShell and there aren’t many products where PowerShell isn’t making inroads into the management structure.

Kudos to Jeffrey Snover and the awesome management technologies team for really selling this internally at Microsoft.

Another point made on the podcast was that Microsoft needed to do more to encourage better development practices… Can those same developers say that their products encourage better application management practices?

Now, we as the users of PowerShell need to step up and convince demand better administrative tooling from our vendors and internal development staffs.  Companies like Quest, VMWare, Idera,  Compellent, and others have gotten the message, but there are still many, many other products out there and many internal applications that suffer from inflexibility.

Web interfaces and GUI tools are nice and can be considered the icing on the cake.  A true manageable application allows for consistent and repeatable actions in an easy to maintain structure, as well as providing flexibility to integrate other potential solutions.  PowerShell provides a lot of that right in the box and allows administrators to bridge the gap and create their own solutions that might not have been supported yet (ever hear – “it’s in the next version”).

So, here is the call to action:

Rise up and demand proper administrative interfaces.

Talk to your managers about the benefits of streamlined application management using a consistent interface across multiple platforms and applications.

Take a developer to lunch and explain how you want to help make using his product a better experience from the application management side.

Let’s take our cue from the oft repeated concept in that Herding Code podcast – there is a need for candid feedback and it is all about the tools that we have to live in and work with every day.

If by chance one of the guys from the Herding Code podcast (or any other developer-centric podcast like .NET Rocks, Deep Fried Bytes, or StackOverflow) happens onto this post and wants to talk further, I’m available.  There are also a good number of PowerShell MVPs and community bloggers who I’m sure would love to provide some “candid feedback” to “developers in the trenches doing real development”.

Larry Clarkin asked me back on the Thirsty Developer to continue talking about development and PowerShell.  We talked about creating cmdlets, hosting PowerShell, and a bit about Version 2.  Check it out here.

I was tagged by SQL Server Expert Brent Ozar in his response to a great, thought provoking blog post called Give Me a Coconut and Six Months by Tim Ford (SQLAgentMan on Twitter).

The short summary of the post is if you had six months free of distraction, what would you turn your attention to.  Tim’s choices included backups, security, and monitoring, which I think is a great “solid foundation” to work from.

Brent posits that if he became more effective at data mining, he would be able to provide a business with critical insight with which to improve sales, product focus, and develop key personnel.

If I had more time (and skills), I could tell executives things like:

• These are the top five customers who are about to leave us.
• These are the top five products that are about to go viral, and we need to stock more ASAP.
• These are the top five salespeople who need coaching to produce more revenue.

Walk into an executive’s office with this kind of information, and you’re a hero.

Here’s the shocker Brent… I agree. 

I agree that delivering that kind of data to management is the Holy Grail of IT projects.  Before coming into the IT realm, I ran a small business and I would have killed for information along those lines. 

One thing I think is missing from Brent’s discussion is that there are three parts to this type of data mining:

• The first part is the technical aspect on how to retrieve the data from databases and other sources of information, which Brent probably has handily covered. 
• The second part is not a technical question at all.  The second part of the data mining is actually coming up with questions you have for your data.  The quote above from Brent’s post highlights those, and point to an area of expertise outside the technical.  Brent is demonstrating (and probably should have said explicitly) that a domain/business knowledge is very important in determining these requests.  This is often difficult for the stereotypical IT person, but is essential if you are interested in moving your career past a technical implementer. 
• So, it appears Brent’s real goal is the the third part of this process, which is translating these questions (taking the business knowledge) and retrieving and correlating the data (using his technical skills) that he will need to make these determinations.

Brent continues on to cover a common area of disagreement between us.

I kick the PowerShell horse a lot, and here it comes again. If you’re in IT, listen up: you’re either cutting costs, or making money.  Guess which one has more upside.  If you truly bust your hump, become an amazing scripting deity, and save 99% of your time, you just saved 99% of your salary.  If you’re really good, you might save 10 people 99% of their time.

I work as the sole admin/accidental DBA/desktop support/multimedia support/”if it has a blinking LED light” support for an agency that collects lots and lots of data.  As a scripting practitioner, if I save myself 25%, 40%, or even 60% of my time not having to solve the same problems over and over, I’m free to plume the mysteries of my database and convert the bits stored their into meaningful data and even information that is usable and actionable. 

Not every environment has the luxury of being able to afford someone of Brent’s caliber to come in and learn their business and help them develop methods of turning their data into knowledge.  In my situation and that of many small to medium businesses, scripting is the tool that enables the IT generalist to explore and branch out into these other areas.

Brent may feel ” you can go into data mining and make 100 salespeople twice as effective” and that IT people who keep things running are replaceable and he may be right, but I believe, especially in tighter economic times, specialists become a luxury that only a few can afford where efficient generalists that know the business become more in demand.

Don’t be mad Brent… it’s okay to be wrong every once in a while!

So if I had six months of no interruptions and could focus on certain specific projects, I would:

• Finish automating (via PowerShell, SQL, or other means) common tasks
• Develop the questions my administrators (not technical admins, but departmental administrators) would like to ask our data sources
• Build tools to answer those questions via reports, graphs, network maps, or other data visualization techniques (like the cool stuff Doug Finke and Chad Miller have been doing)

I personally think that EVERY technical person should have a grasp on the basic business environment, be aware of what is happening in their company’s industry (or at least their department’s industry for larger organizations), and begin to develop more in depth domain knowledge as to the business processes.

Due to the odd path I took to becoming an IT person, I’ve actually accumulated quite a bit of domain knowledge about the law enforcement, the various jobs, information requirements, and the ins and outs of our workflows and data.  This background gives me a great starting place when going to look at my data, since scripting has given me the time to do so. 

Judging from the pattern in Tim’s and Brent’s posts, I’m supposed to tag a few people..

How about:

• Doug Finke (developer, data visualization explorer,  and PowerShell MVP),
• Hal Rottenberg (administrator, VMWare vExpert, podcaster, and PowerShell MVP),
• and Wes Stahler (administrator, blogger, and PowerShell enthusiast)

The PowerShell paradigm is the task based cmdlet.  With cmdlets that surface a single function that handle a multitude of inputs, a PowerShell session or script can read back like a sentence. 

Get-Process –Name iexplore | Where-Object {$_.WorkingSet –gt 50000000 } | Stop-Process

In the above (often overused, but illustrates the point well) example, Get-Process, Where-Object, and Stop-Process all accurately describe the task that they perform.

We can take this same approach with our scripts and functions.  By keeping your scripts and functions focused on performing discreet tasks, you keep with the composable nature of PowerShell and provide yourself the most flexibility in reusing the functions you create.

One of the most common “errors” I see in PowerShell scripts that are shared is the embedding of formatting in the output of the script.  If your script generates output, it should be in the form of objects (custom or otherwise).  (The only caveat here is if your script’s functionality is to handle the output of data, in which case it’s sole focus should be the handling that output.)

The PowerShell ecosystem provides a number of tools for formatting the output of your scripts, whether you want to display your scripts in the console window (Format-List, Format-Wide, Format-Custom, Format-Table), as a web page (ConvertTo-HTML), sent to a CSV file (Export-CSV) or XML (Export-CLIXML), in Version 2 to a sortable, groupable grid (Out-GridView), as well as a number of other options (Visifire graphs via PowerBoots, Excel, PDF, or Image via the Reporting Services Redistributable, or as a netmap using NodeXL).

Take advantage of these, or write your own.  If you write your own, it should work with custom objects, to keep with the composable nature of PowerShell.

DRY?  Does this mean I can’t use PowerShell on a water-cooled PC?

DRY is a principle that can be very familiar to the PowerShell aficionado with a development background.  DRY means Don’t Repeat Yourself.  Keeping your scripts DRY means that our scripts don’t contain repeated code.  Copy/Paste is not your friend!

Why should PowerShell scripters care about keeping their PowerShell DRY?  One major reason – script maintainability.

PowerShell has a huge advantage over scripting environments/shells in that the noun/verb structure lends itself to very readable scripts.  If there is duplication in your code, that readability can give a PowerShellers a bit of overconfidence when reading/modifying scripts that are new to them or that have not been looked at in a while.

If you find yourself copying and pasting lines of PowerShell between different sections of your script, you could be setting yourself up for some interesting times when you need to make a change.

For the next month or two, the PowerScripting Podcast will be live on Tuesday nights at 9 PM Eastern / 8 PM Central.  Be sure to update your calendars and stop in for some great, interactive PowerShell content.

One concept that is central to the understanding of the layout of classes in the Base Class Libraries is the Namespace.  Namespaces are a method of providing context for classes (and other constructs like Enums and Structs), allowing developers to group related classes and not worry about name collisions with development in other areas.

In the Base Class Libraries, the root namespace is the System namespace.  The System namespace defines a large amount of the base types, like Object (which all classes derive from), String, DateTime, Boolean, and numerous others.  When you specify types, PowerShell can infer the “System” part.

Example:

PS C:\scripts\PowerShell> $random = New-Object –TypeName Random

is the same as

PS C:\scripts\PowerShell> $random = New-Object –TypeName System.Random

Namespaces are hierarchical, except in naming convention. The representation of a file in PowerShell is a great example.  A file object is represented in PowerShell as a System.IO.FileInfo object.  The FileInfo class lives in the System.IO namespace, which is a separate namespace than the System namespace.  There is a hierarchical impression implied, but that is solely through naming convention.

The CLR (Common Language Runtime) does not care about namespaces, only the fully qualified name of the class, so why use namespaces at all?

Namespaces are a convenience for developers, as languages like C# and VB.NET have language constructs that allow the developer to not have to use the fully qualified type name every time that is needed to be referenced.  C# provides the “using” syntax and VB.NET has the “imports” syntax.

How does this apply to PowerShell?

PowerShell does not currently have a “using” or “imports” syntax or language feature, but there are a couple of workarounds.  You can use variables holding the namespace and string concatenation to save some typing

PS C:\scripts\PowerShell> $netinfo = ‘System.Net.NetworkInformation’
PS C:\scripts\PowerShell> $ping = New-Object “$netinfo.ping”
PS C:\scripts\PowerShell> $ping.gettype()
IsPublic IsSerial Name           BaseType
——– ——– —-                ——–
True     False    Ping            System.ComponentModel.Component

This doesn’t work with type accelerators though (at least in V1)..  V2 CTP3 has some other options and Oisin Grehan has a great discussion on how to find the type accelerators and add your own.

Another option would be to create a function to wrap the creation of objects for particular namespaces.

Use-Namespace is a function I came up with to create a function to wrap New-Object for specific namespaces.

The useage of Use-Namespace is
PS C:\scripts\PowerShell>Use-Namespace System.IO, System.Net
PS C:\scripts\PowerShell>$memorystream = New-System.IOObject MemoryStream
PS C:\scripts\PowerShell>$memorystream.GetType()
PS C:\scripts\PowerShell> $memorystream.gettype()
IsPublic IsSerial  Name                    BaseType
——– ——–   —-                         ——–
True     True     MemoryStream      System.IO.Stream

And the function is here..

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

function Use-Namespace()
{
  param ([string[]]$Namespace)
  BEGIN
  {
    if ($Namespace.length -gt 0)
    {
      $Namespace | Use-Namespace
    }
  }
  PROCESS
  {
    if ($_ -ne $null)
    {
      $NS = $_
      $NSObject = "$NS" + "Object"
      $function = @"
param (`$Class, [string[]]`$ArgumentList)
if (`$ArgumentList.length -gt 0)
{
return New-Object -TypeName "$NS.`$Class" -ArgumentList `$ArgumentList
}
else
{
return New-Object -TypeName "$NS.`$Class"
}
"@
    Set-Item -Path "function:global:New-$NSObject" -Value $function -force
    Write-Debug "Created function:global:New-$NSObject "
    }
  }
}

PSMDTAG:FAQ CLR
PSMDTAG:FAQ Namespace
PSMDTAG:FAQ .NET Framework

This morning, Scott Herold and I did back to back presentations on PowerShell and the Virtualization EcoShell for the Wisconsin VMWare Users Group.

Time flew by and I didn’t get to the second half of my slide deck, so if you are interested, here it is.

If you want to see the webcast, you can find it here.  Just a word of warning, I haven’t listened to it yet, but we were recording over Skype and a fixed microphone while speaking over another mic in a larger room.  I’m guessing the sound quality is not the best, but should be understandable.

Thanks to Scott Herold for sharing the “stage” with me and thanks to Rod Gabriel and the WI VMUG for allowing me to share a bit about PowerShell.

UPDATE: The script was moved to Google Code.  The links in the post have been updated to reflect that.  Or you can just go here… http://code.google.com/p/poshcodegen/

I’ve been working on some data conversion at work, converting records from one system to a new system.  I’ve built quite a library of SQL queries with PowerShell wrappers for dealing with data in the first system, but I don’t have the same luxury with the new system.

The new system does, however, have a nice set of stored procedures that make moving data into their application much easier.

I started writing my conversion scripts in PowerShell, since I do have to do some processing on the records to accommodate the new workflow and data layout.  I was looking at having to call almost 100 stored procedures through various parts of this process.  That is a lot of boiler plate code or referring back to the database often to check parameter names and types.  So, I’ve written a little PowerShell script that will take stored procedures (either as a parameter or from the pipeline) and create a function that wraps that stored procedure.

The benefits of this are great with V2 CTP3 or with a more advanced editor (one that will provide tab completion on parameters).

One a wider scope, I think that this type of utility is one of PowerShell’s great strengths.  Using PowerShell for metaprogramming (another example here on the Telling Machine blog) can be a great time saver.  I spent a couple of hours working on this script, but it would have cost me much more time to handle each case individually.

Now, when I hear “metaprogramming”, my head starts to hurt a bit as I start to think about programs about programs about programs, but this isn’t that bad.  PowerShell makes this pretty easy to understand though.  To create a function dynamically, all that is needed is a string that contains text that the PowerShell runtime can evaluate (PowerShell will check for syntax errors, but not logic errors – as is the case with any script or function).

Example:

$text = ‘Get-ChildItem *.ps1 | Measure-Object’

Set-Item –Path function:global:Get-PowerShellScriptCount –Value $Text

This takes advantage of the Function provider and creates a function object in the global scope with the specified name and $Text is turned into a scriptblock.  I can then call that function as needed.

Since I know PowerShell, to build dynamic functions I just have to create text that can be evaluated to do the function I need.

Here’s what my script does after it runs a query against the database to get the stored procedure’s text:

1. parses the text to get the parameter names and types
2. using the names and types, it sets the parameters for the function (if someone wants to add some logic to make it type safe, that would rock!)
3. builds the text to create a SqlConnection object to handle the database connection
4. builds the text to create a SqlCommand object and sets the type of command to be a stored procedure
5. builds the text to populate the parameters, including setting up the output parameters (if any)
6. builds the text to run the stored procedure
7. builds the text to put any output parameters into a PSObject as properties.
8. create a new function with the name of the stored procedure and uses the text built in the previous steps as the scriptblock for the function.

What are you automating with PowerShell?

How about trying to automate some of your automation code?

New-StoredProcFunction.ps1 here.

PSMDTAG:metaprogramming sql stored procedure

UPDATED SCRIPT: Thanks to Chad Miller for the idea.. instead of parsing the text of the stored procedure, the parameter information is available in the Information_Schema.Parameters

On May 19th, Scott Herold and I will be presenting for the Wisconsin VMWare Users Group. 

I’ll be covering the basics of PowerShell, as well as an introduction to the VI Toolkit (for Windows)/PowerCLI.

Scott will be talking about  the Virtualization EcoShell Initiative (VESI).

If you are in Wisconsin, the meeting details can be found here.

If you are not able to make the meeting, VMWare will be providing a live stream of our presentations via WebEx, as well as providing recordings after the fact.  

My presentation starts at 9 AM Central time and Scott follows me at about 9:50.

This year at TechEd (North America), Hal Rottenberg and I will be there promoting PowerShell and PowerShellCommunity.Org.  We will be bouncing around various sessions, the PowerShell booth, and the remaining conference.

If you have a PowerShell story that you want to share, find me (I’m a big guy and easy to pick out of a crowd).  I’ll have some recording gear and be looking to collect some user stories.

See you there!

I really enjoy Jeffery Hicks’s Prof. PowerShell column.  In two of his recent columns (Perfect Properties and Label Me Perfect), Jeffery goes in to “calculated properties”.  

A calculated property is a hashtable passed to the Select-Object or Format-* cmdlets as one of the properties to return.

Here is an example from Label Me Perfect:

PS C:\> get-wmiobject Win32_logicaldisk -filter “drivetype=3″ `
-computer “mycompany-dc01″ | Format-table DeviceID,VolumeName,`
@{label=”Size(MB)”;Expression={$_.Size/1mb -as [int]}},`
@{label=”Free(MB)”;Expression={$_.FreeSpace/1MB -as [int]}},`
@{label=”PercentFree”;Expression={
“{0:P2}” -f (($_.FreeSpace/1MB)/($_.Size/1MB))}} -autosize

That is a whopper to type at the command line or to try to read in a script.

In order to increase readability and resuse, you can assign those hash tables to a variable and use that variable as one of the parameters to Select-Object or one of the Format-* cmdlets.

PS C:\> $Size = @{label=”Size(MB)”;Expression={$_.Size/1mb -as [int]}}
PS C:\> $FreeMB = @{label=”Free(MB)”;Expression={$_.FreeSpace/1MB -as [int]}}
PS C:\> $PercentFree = @{label=”PercentFree”;Expression={”{0:P2}” -f (($_.FreeSpace/1MB)/($_.Size/1MB))}}

PS C:\> get-wmiobject Win32_logicaldisk -filter “drivetype=3″ `
-computer “mycompany-dc01″ | Format-table DeviceID,VolumeName, `
$Size, $FreeMB, $PercentFree -autosize

This makes it a bit easier to read, and also makes those hash tables available for further use in additional statements without needing to retype them.

PSMDTAG: FAQ: Calculated Properties

Previously, I published a script on comparing what tables two databases contained.  Going a bit further, I put together a script that compares the columns and what type of data they store.

Compare-DatabaseColumns has similar parameters to the Compare-DatabaseSchema script.

• Table – One or more tables to compare columns from
• SqlServerOne – SQL Server for the first database
• FirstDatabase – Name of the first database for the comparison
• SqlUsernameOne – SQL user name for the first database
• SqlPasswordOne – SQL password for the first database
• SqlServerTwo – SQL Server for the second database
• SecondDatabase – Name of the second database for comparison
• SqlUsernameTwo – SQL user name for the second database
• SqlPasswordTwo – SQL password for the second database
• FilePrefix – Prefix for the log file name
• Log – Switch parameter that saves one CSV file with the difference in the tables.  If the Column switch parameter is chosen also, it will save one CSV file per table with differences in the columns

This script can also take pipeline input, either strings or a property of “Name” or “TableName”.

You can find this script on PoshCode.org.

On April 1st, 2009, Microsoft is putting on a free 24 hour virtual event covering developer-related topics.
There will be 95 live sessions provided via Live Meeting in the following tracks:

• Windows Development and Frameworks
• Windows Mobile Development
• Office and SharePoint Development
• Developer Tools, Languages & Practices
• Web Development & User Experience

PowerShell Community’s own Marco Shaw (MVP) will be presenting the ONLY PowerShell session:

WIN300 Scripting the Microsoft .NET Framework Using Windows PowerShell
Now that PowerShell is part of the Microsoft Common Engineering Criteria, expect to see more and more PowerShell in Microsoft server products. This session looks at how PowerShell can be used to directly access the .NET Framework. Some simple examples are demonstrated, but also more advanced examples of using Windows Presentation Foundation (.NET 3.0) and LINQ (.NET 3.5) are discussed.

You can find more information about this event here or on Marco’s blog. 

I’m looking forward to seeing this presentation.  Hope to “see” you there!  Register now..