Tuesday, 13 December 2016

Moveing classic resources (within subscription)

Well – according to the documentation then you can use the portal to move classic resource items around your groups.

“

  • Virtual machines (classic) must be moved with the cloud service.
  • Cloud service can only be moved when the move includes all its virtual machines.”

…

“To move classic resources to a new resource group within the same subscription, use the standard move operations through the portal, Azure PowerShell, Azure CLI, or REST API. You use the same operations as you use for moving Resource Manager resources.”

It even has a nice little picture showing it being done

I had a group with two resources – a VM and its cloud service – but the portal would not allow them to be moved.

So this nice little script does it

# prompt for credentials etc.
Login-AzureRmAccount
# select the correct subscription
Get-AzureRmSubscription -SubscriptionName "Stiona Software General" | Select-AzureRmSubscription
# get the ResourceID property of the two resources - luckily they have the same name in my case
$resources = Get-AzureRmResource -ResourceName fusionreports -ResourceGroupName fusionreports | Select -ExpandProperty ResourceId
# issue a move to the new group
Move-AzureRmResource -DestinationResourceGroupName fusionazure -ResourceId $resources

The $resources this -

/subscriptions/b9fee249-e903-4c4a-ade7-42982be9a20f/resourceGroups/fusionreports/providers/Microsoft.ClassicCompute/domainNa
mes/fusionreports
/subscriptions/b9fee249-e903-4c4a-ade7-42982be9a20f/resourceGroups/fusionreports/providers/Microsoft.ClassicCompute/virtualM
achines/fusionreports

After promt to move the old resource group was empty and the new one contained the vm and cloud service.

Nice hint from stack overflow on the use of  ExpandProperty parameter of Select-Object for getting an array of values from an array of objects.

Man – working with Azure classic sure is tough.

Monday, 5 December 2016

Finding dependencies

Looking to find service references and database connections in web.config files on a server or a set of servers.

I started with this guys script - http://www.markrainey.me/2013/03/finding-urls-in-text-files.html but he was just interested in unique url’s so I hacked it around a bit and also added in another regex for the connection strings.

This is not perfect by any means but is a start for anyone else (or me in the future)

########################################################### 
# AUTHOR  : Mark Rainey  
# Ammended - Stuart McLean 2016-05-12
# DATE    : 2013-03-13  
# COMMENT : Reads in a list of servers and searches
# the E:\Live folder for config files.  When it finds
# them it searches for anything starting with 3 or more
# letters (tcp or http) followed by a colon and \\
# now also outputs connection string data and 
# looks for </value in urls
###########################################################

#ERROR REPORTING ALL
Set-StrictMode -Version latest

$scriptpath = Split-Path -parent $MyInvocation.MyCommand.Definition
# A friend at work helped me get this to output to Excel
$outputFile = $scriptpath + "\urls.csv"
$connectionFile = $scriptpath + "\connections.csv"
# This is a file with a FQDN of each server on a new line
$serverList = $scriptpath + "\serverlist.txt"
# Load server list
$servers = Get-Content $serverList


$URLS = New-Object System.Collections.ArrayList
$ConnectionStrings = New-Object System.Collections.ArrayList

$credential = Get-Credential

# Find the string and save it to a file
Function getStringMatch
{
 Try { 
  # Loop through all servers
  Foreach ($server In $servers) {
      # Set UNC Path to files for this server
   $drive = "\\" + $server + "\d$"
   New-PSDrive -Root $drive  -PSProvider FileSystem -Name z -Credential $credential
   #The path can be anywhere on your servers you want to search
   $path     = "z:\AppWebSites"
   #Get list of files
   $files    =  Get-Childitem $path web.config -Recurse | Where-Object { !($_.psiscontainer) } 
   # Loop through the server and search all config files under E:\Live
    #$matches = New-Object System.Collections.ArrayList
   #$matches.clear()
   Foreach ($file In $files)
      {
    $fullFileName = $file.FullName 
    # regular expression for a URL format
    #$regex = '([a-zA-Z]{3,})://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)*?'
    # find any urls
    $regex = '([a-zA-Z]{3,})(:\/\/.*)(?=<)'
                # Find all matches in current file and add the Value for each one to an array
    select-string -Path $fullFileName -Pattern $regex -AllMatches | % { $_.Matches } | % {
        $found = New-Object psobject -Property  @{       
        server = $server;
        urlRef = $_.Value;
        file = $file.FullName
        }
        
     $URLS.add($found)
    }

    $connectionStringRegex = '(?<=connectionString=")(.*)(?=")'
    select-string -Path $fullFileName -Pattern $connectionStringRegex -AllMatches | % { $_.Matches } | % {
    try{
        $connectionString = $_.Value;

        $source =   ($connectionString | select-string -Pattern '(?<=data source=)(.*?)(?=;)').Matches[0].Value 
        $catalog =  ($connectionString | select-string -Pattern '(?<=Initial Catalog=)(.*?)(?=;)').Matches[0].Value 
        $sqlUser =  ($connectionString | select-string -Pattern '(?<=User ID=|uid=)(.*?)(?=;)').Matches[0].Value 
        $found = New-Object psobject -Property  @{       
        server = $server;
        source = $source;
        catalog = $catalog;
        sqluser = $sqlUser;
        file = $file.FullName
        }
        
     $connectionStrings.add($found)
     }
     Catch {
  Write-host "could not parse connection string $_" + $connectionString
 }
      } 
   }
            
   Remove-PSDrive z
  
  }
  
 }
 Catch {
  Write-host "Something failed $_"
 }
 Finally {
  "Finished"
 }
 
}
  getStringMatch
  $URLS.ToArray() | Export-Csv -Path $outputFile -NoTypeInformation
  $connectionStrings.ToArray() | Export-Csv -Path $connectionFile -NoTypeInformation

Wednesday, 30 November 2016

Powershell file compare.

Simple script for comparing files in two directories.

param (
       [Parameter(Mandatory=$true)][string]$originalDir,
        [Parameter(Mandatory=$true)][string]$newDir
)

$fso = Get-ChildItem -Recurse -path $originalDir
foreach($file in $fso.Name) {
echo comparing $originalDir\$file to $newDir\$file
Compare-Object -ReferenceObject $(Get-Content $originalDir\$file)  -DifferenceObject $(Get-Content $newDir\$file) -IncludeEqual
}

Will compare line by line the files in the originalDir with those in newDir with the same name. 

  • Probably breaks if the file isn’t in newDir
  • Doesn’t check for extra files in newDir.

Tuesday, 22 November 2016

Connection IP’s to my database

Thanks to @j_mcmullan

Apperently there's a lot more info on these tables

 select distinct--c.session_id
	  c.client_net_address
	  ,s.database_id
	  ,db_name( s.database_id)
from sys.dm_exec_connections c
    inner join sys.dm_exec_sessions s on c.session_id = s.session_id
where s.database_id = db_id('ASPState')
GO

(I beleiveit may be ironical that I was looking at sessions to a session database).

Friday, 18 November 2016

Sprint 4–put it live.

Not quite as easy as deploying the sql as connecting locally I’ve been using my own credentials.  For an azure connection I need to:

  1. Create a database role and grant exec on the stored procs to that role.
  2. Create a new database login
  3. Put the database login in the role.

Why don’t I just use the “sa” password.

Well – least privilege – a good security practice.

Sprints 2 and 3 retrospective–an hour really isn’t long.

So – sprint 2 took about an hour and a half of coding – so a lot closer than the first.

I then noticed that I still had some inline nasty SQL in my powershell – so I cracked out another hour and got another proc and changes in.

Main lesson so far – an hour is a really short time – keep pebbleising the tasks.

Outcome

Ok – so the work got done and hopefully I’ll have a list of riders on production soon.

 

I’ve thought hard about it and its definitely a useful task towards the end goal as the first “Minimum” viable product is for riders to sign up.  I’m going to allow google authentication/Oath which will strongly (enough for this) validate the users email which I can then use to look up there details in the data I have.

 

 

A good start.

 

Some feedback on the tasks here – really putting this code here for anyone to look @ until I work out if/what to open source.  Unless you particularly want to critique my sql or look at how I use powerhell to call stored procs then not much excitement.

 

“Work out how to deploy database project to local database and how to connect to it. “

this was pretty easy – just set the PowerShell script to depend on the database project and then it deploys all your changes as it builds – pretty nifty stuff

 

“Create stored procs for crud operations required by script – include find rider.”

CREATE PROCEDURE [dbo].[ups_RiderInsert]
    @vcrFirstName    NVARCHAR (100) ,
    @vcrSurname      NVARCHAR (100) ,
    @vcrAddressLine1 NVARCHAR (100) ,
    @vcrAddressLine2 NVARCHAR (100) ,
    @vcrTown         VARCHAR (100)  ,
    @vcrCountryCode  VARCHAR (2)    ,
    @vcrCountry      VARCHAR (100)  ,
    @vcrPhoneNumber  VARCHAR (20)   ,
    @vcrClub         VARCHAR (50)   ,
    @vcrClubNumber   VARCHAR (20)   ,
    @vcrECName       VARCHAR (100)  ,
    @vcrECTelephone  VARCHAR (100) 
AS
SET XACT_ABORT ON

BEGIN TRANSACTION
	INSERT INTO tblRider
	VALUES( 
	@vcrFirstName,
	 @vcrSurname      ,
    @vcrAddressLine1 ,
    @vcrAddressLine2 ,
    @vcrTown           ,
    @vcrCountryCode  ,
    @vcrCountry        ,
    @vcrPhoneNumber  ,
    @vcrClub            ,
    @vcrClubNumber      ,
    @vcrECName       ,
    @vcrECTelephone)

	SELECT SCOPE_IDENTITY()  AS INT
COMMIT TRANSACTION



CREATE PROCEDURE [dbo].[usp_EventParticipentInsert]
    @intRiderId       INT  ,
    @intEventId       INT  ,
    @decPaidSTG       MONEY,
    @decPaidEUR       MONEY
AS
SET XACT_ABORT ON
BEGIN TRANSACTION
INSERT INTO tblEventParticipant
VALUES(
   @intRiderId       ,
    @intEventId      ,
    @decPaidSTG      ,
    @decPaidEUR      
)
	SELECT SCOPE_IDENTITY()  AS INT

COMMIT
CREATE PROCEDURE [dbo].[usp_LicenceGetByRiderAndYear]
	@intRiderId int,
	@intYear int
AS
	SELECT * FROM dbo.tblLicense 
	WHERE
	intRiderId = @intRiderId
	AND intYear = @intYear
CREATE PROCEDURE [dbo].[usp_LicenseInsert]
    @intYear          INT,
    @intRiderId       INT,
    @vcrLicenceNumber NVARCHAR(20)

AS
SET XACT_ABORT ON
BEGIN TRANSACTION
INSERT INTO tblLicense 
VALUES(@intYear , @intRiderId, @vcrLicenceNumber)
COMMIT
CREATE PROCEDURE [dbo].[usp_RiderSearchByName]
	@FirstName nvarchar(100) ,
	@SurName nvarchar(100)
AS
	SELECT * FROM tblRider r
	WHERE (r.vcrFirstName like @FirstName + '%' OR @FirstName IS NULL)
	AND (r.vcrSurname like @SurName + '%' OR @SurName IS NULL)
	

 

 

    “Enahance script / tidy up for stored proc use and other comments in script below.”

param (
	   [Parameter(Mandatory=$true)][string]$fileName,
		[Parameter(Mandatory=$true)][string]$connString,
		[Parameter(Mandatory=$true)][int]$intEventId,
	[Parameter(Mandatory=$true)][int]$intYear
)


function ToDBNull($var) {
	if($var -eq $null) {return [System.DBNull]::Value}
	return $var;
}

#
# Script.ps1
# TODO - rename the bit above and document this method
function parseExcel([String] $fileName){
	$excel=new-object -com excel.application
	$wb=$excel.workbooks.open($fileName)
	$sh = $wb.Sheets.Item(1)
	# TODO de hard code teh endrow.
	$startrow = 2
	$endrow = 16#Connection and Query Info

# prpbably best to pass the database name etc. through as Parameters

$conn = new-object System.Data.SqlClient.SqlConnection $connString 
	#TODO Should the open be in the try/catch?
$conn.Open()
try{

	# Main for loop for parsing sheet
	# TODO rather than hard coding rows do until no end.

	for($i = $startrow;
		($sh.Cells.Item($i,1).Value2 -ne $null) -and ($sh.Cells.Item($i,1).Value2.Trim() -ne "") ; 
		$i++) {
$Name = $sh.Cells.Item($i,1).Value2
$Surname = $sh.Cells.Item($i,2).Value2
[String]$Address = $sh.Cells.Item($i,3).Value2
$Tel = $sh.Cells.Item($i,4).Value2
$email = $sh.Cells.Item($i,5).Value2
$Club = $sh.Cells.Item($i,6).Value2
$CI = $sh.Cells.Item($i,7).Value2
$ClubNo = $sh.Cells.Item($i,8).Value2();
$EC = $sh.Cells.Item($i,9).Value2
$ECTel = $sh.Cells.Item($i,10).Value2
$PaidE = $sh.Cells.Item($i,11).Value2
$PaidS = $sh.Cells.Item($i,12).Value2 
		# TODO - do we need to close the command - check documentation 
		# to see if it implements IDisposable - and is there a using equivalent in PowerShell
[System.Data.SQLClient.SQLCommand]$Command = New-Object System.Data.SQLClient.SQLCommand

	$Command.Connection = $conn
			$Command.CommandType = [System.Data.CommandType]::StoredProcedure
			# - lookup first - do within proc?.

	$Command.CommandText = "[dbo].[usp_RiderSearchByName]"
	
	$Command.Parameters.aDD("@FirstName",[System.Data.SqlDbType]::NVarChar)
	$Command.Parameters["@FirstName"].Value = $Name
	$Command.Parameters.Add("@SurName",[System.Data.SqlDbType]::NVarChar ).Value = $Surname
	[int]$ID = $null
   [System.Data.SqlClient.SqlDataReader ]$reader = $Command.ExecuteReader()
			try{
				if($reader.Read()){
					$ID = $reader["intRiderID"]
				}
			}
			finally{
				$reader.Close()
			}
			# if we didn'te find a rider then create on
			if($ID -eq 0) {
				[System.Data.SQLClient.SQLCommand]$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

			$Command.Connection = $conn
			$command.CommandType = [System.Data.CommandType]::StoredProcedure

				$Command.CommandText = "[dbo].[ups_RiderInsert]"
				$Command.Parameters.Add("@vcrFirstName",[System.Data.SqlDbType]::NVarChar ).Value = $Name
				$Command.Parameters.Add("@vcrSurname",[System.Data.SqlDbType]::NVarChar ).Value = $Surname
				$Command.Parameters.Add("@vcrAddressLine1",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($Address.Split(',')[0])
				$Command.Parameters.Add("@vcrAddressLine2",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($Address.Split(',')[1])
				$Command.Parameters.Add("@vcrTown",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($Address.Split(',')[2])
				$Command.Parameters.Add("@vcrCountryCode",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($null)
				$Command.Parameters.Add("@vcrCountry",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($null)
				$Command.Parameters.Add("@vcrPhoneNumber",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($Tel)
				$Command.Parameters.Add("@vcrClub",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($Club)
				$Command.Parameters.Add("@vcrClubNumber",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($ClubNo)
				$Command.Parameters.Add("@vcrECName",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($EC)
				$Command.Parameters.Add("@vcrECTelephone",[System.Data.SqlDbType]::NVarChar ).Value = ToDBNull($ECTel)
		
				[int]$ID = $Command.ExecuteScalar()
				}
	
[System.Data.SQLClient.SQLCommand]$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

	$Command.Connection = $conn
			$command.CommandType = [System.Data.CommandType]::StoredProcedure
	$Command.CommandText = "[dbo].[usp_EventParticipentInsert]"
				$Command.Parameters.Add("@intRiderId",[System.Data.SqlDbType]::Int).Value = $ID
				$Command.Parameters.Add("@intEventId",[System.Data.SqlDbType]::Int).Value = $intEventId
					$Command.Parameters.Add("@decPaidSTG",[System.Data.SqlDbType]::Money).Value = If($PaidS -ne $null)  {$PaidS} else {0}
					$Command.Parameters.Add("@decPaidEUR",[System.Data.SqlDbType]::Money).Value = If($PaidE -ne $null)  {$PaidE} else {0}


		

	$Command.ExecuteScalar()
			# if got a CI number
			if($CI -ne $null -and $CI.Trim() -ne '') { 
# search for a licence

			[System.Data.SQLClient.SQLCommand]$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

			$Command.Connection = $conn
			$command.CommandType = [System.Data.CommandType]::StoredProcedure
			$Command.CommandText = "[dbo].[usp_LicenceGetByRiderAndYear]"
			$Command.Parameters.Add("@intRiderId",[System.Data.SqlDbType]::Int).Value = $ID
			$Command.Parameters.Add("@intYear",[System.Data.SqlDbType]::Int).Value = $intYear
	
			[int]$LID = $null
   [System.Data.SqlClient.SqlDataReader ]$reader = $Command.ExecuteReader()
			try{
				if($reader.Read()){
					$LID = $reader["intRiderID"]
				}
			}
			finally{
				$reader.Close()
			}
				# no licence found - create one
	if($LID -eq 0){
	[System.Data.SQLClient.SQLCommand]$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

			$Command.Connection = $conn
			$command.CommandType = [System.Data.CommandType]::StoredProcedure
			$Command.CommandText = "[dbo].[usp_LicenseInsert]"
			$Command.Parameters.Add("@intRiderId",[System.Data.SqlDbType]::Int).Value = $ID
			$Command.Parameters.Add("@intYear",[System.Data.SqlDbType]::Int).Value = $intYear
	$Command.Parameters.Add("@vcrLicenceNumber",[System.Data.SqlDbType]::NVarChar).Value = $CI
		$Command.ExecuteScalar()
				}
	

	}

	}
}
Finally
{
$conn.Close()
}
$wb.Close()
$excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)

	}


	parseExcel($fileName)

Thursday, 10 November 2016

Sprint 2

Goal

Refactor the import script so it is fit for purpose.

Tasks

  1. Work out how to deploy database project to local database and how to connect to it.
  2. Create stored procs for crud operations required by script – include find rider.
  3. Enahance script / tidy up for stored proc use and other comments in script below.

Outcome

I can parse all the sheets I have to load an initial dataset.  I have a set of re-useable stored procedures for the application.

 

The script to fix

#
# Script.ps1
# TODO - rename the bit above and document this method
function parseExcel([String] $fileName){
	$excel=new-object -com excel.application
	$wb=$excel.workbooks.open($fileName)
	$sh = $wb.Sheets.Item(1)
	# TODO de hard code teh endrow.
	$startrow = 2
	$endrow = 16#Connection and Query Info

# prpbably best to pass the database name etc. through as paramaters
$serverName="uj6arxo0jh.database.windows.net" 
$databaseName='AudaxDB' 
#TODO  And this really shouldn't be here!
$connString = "Server=$serverName;Database=$databaseName;" 
$conn = new-object System.Data.SqlClient.SqlConnection $connString 
	#TODO Should the open be in the try/catch?
$conn.Open()
try{

	# Main for loop for parsing sheet
	# TODO rather than hard coding rows do until no end.

	for($i = $startrow; $i -le $endrow; $i++) {
$Name = $sh.Cells.Item($i,1).Value2
$Surname = $sh.Cells.Item($i,2).Value2
[String]$Address = $sh.Cells.Item($i,3).Value2
$Tel = $sh.Cells.Item($i,4).Value2
$email = $sh.Cells.Item($i,5).Value2
$Club = $sh.Cells.Item($i,6).Value2
$CI = $sh.Cells.Item($i,7).Value2
$ClubNo = $sh.Cells.Item($i,8).Value2();
$EC = $sh.Cells.Item($i,9).Value2
$ECTel = $sh.Cells.Item($i,10).Value2
$PaidE = $sh.Cells.Item($i,11).Value2
$PaidS = $sh.Cells.Item($i,12).Value2 
		# TODO - do we need to close the command - check documentation 
		# to see if it implements IDisposable - and is there a using equivalent in PowerShell
$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

    $Command.Connection = $conn
		# TODO - convert to strongly typed stored procedure.
		# also - lookup first - do within proc?.
    $Command.CommandText = "INSERT INTO tblRider VALUES( '" + $Name  + "'" +
    ",'" + $Surname.Replace("'","''") + "'" + # notice the fancy replace here for the O' people
    ",'" + $Address.Split(',')[0] + "'" +
    ",'" + $Address.Split(',')[1] + "'" +
    ",'" + $Address.Split(',')[2] + "'" +
    ",NULL" + 
    ",NULL" +
    ",'" + $Tel + "'" +
    ",'" + $Club + "'" +
    ",'" + $ClubNo + "'" +
    ",'" + $EC + "'" +
    ",'" + $ECTEL + "'" +
        ")`n`r" + 
        "SELECT @@IDENTITY `n`r"
        
        
        $Command
        [int]$ID = $Command.ExecuteScalar()
        $ID
$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand
		# TODO - remove not required
		#TODO create stronly typed proc.
    $Command.Connection = $conn
    $Command.CommandText = 
       "INSERT INTO tblEventParticipant VALUES( " +  $ID  +",1,'" + $PaidS + "','" + $PaidE +"')" 
    $Command
    $Command.ExecuteScalar()

		# TODO again - needs stored proc.
		# 
if($CI -ne $null -and $CI.Trim() -ne '') { 
$Command =  $Command = New-Object System.Data.SQLClient.SQLCommand

    $Command.Connection = $conn
    $Command.CommandText = 
       "INSERT INTO tblLicense  VALUES( 2016, " +  $ID  + ",'" + $CI +"')" 
    $Command
        $Command.ExecuteScalar()

    }

	}
}
Finally
{
$conn.Close()
}
$wb.Close()
$excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)

	}

	parseExcel("C:\Users\stuartm\OneDrive\Documents\Audax\A300_2016\A30002016Entrants.xlsx")