Learning Gulp

Overview

I’m in a learning mode picking up a lot of the ‘new’ stuff I have missed over the past few years.  One area is all the new tooling like bower, yeoman, gulp and so.  To a degree I’m skeptical that I need this stuff.  I like writing PowerShell scripts or MSBuild scripts for most tasks that need to be automated.  All the same: no reason not to at least look.

You can download all of the files from here

What is Gulp?

Gulp is described as a JavaScript task runner.  What that really means is that they’ve build a tool on top of node.js that has plug-ins available that will allow you to work with files and do things to them.  It also includes a file watcher that will run scripts in response to changes to the files it is watching.

The idea behind Gulp is to provide a build for your website.  HTML, JavaScript and CSS on their own don’t require anything to be built, but if you are using any of the languages built on top of them (LESS, SASS, etc.) then you need something to ‘compile’ the file and output your CSS or JavaScript.  You also might what your files minified which requires some processing.

In my case I use Visual Studio which handles nearly all this stuff for me.  But not everybody has such an awesome tool at their disposal.

Get it setup so you can run it from PowerShell

Before going further I’ll point out that none of this is strictly necessary.  Most of the time I work from Visual Studio which will process gulp files without any help.  You could also save yourself some trouble and just fire up the old windows command prompt and live on happily.  However, I’m all about automation.  What if I want to use gulp as a part of an automated workflow I’ve scheduled via Windows Task Scheduler or some other scheduler?  Most of my automation work is done via PowerShell.  If you’ll bear with me a little this might make a little sense.  Or not.

As already mentioned I primarily work from PowerShell so a little additional work is necessary to get things setup.  The first task is to get node and npm on the workstation.  Go to http://nodejs.org/ and install NodeJS (just click on the tile that matches your system).  This will also install npm (node package manager).  To verify that you’re ready to go, do the following:

Start PowerShell and at the command prompt type “node -v” this will show the version of Node.

pic1

Now type “npm -v” to see the version of Node Package Manager:

pic2

We are now ready to install gulp.   To start with we’ll create a project.  In my case I keep stuff like this in a folder called trashcode.  So we’ll create a gulp folder to hold all of the experiments with gulp and then a project in there called concat.  Like so:

pic3

Next we initialize npm by typing “npm -init” and answer the questions:

pic4

Notice that I change the entry point from “index.js” to “gulpfile.js”.  I also provided a description to avoid the warning, if you’re not OCD you can leave description empty.  You now see we have a package.json file in our directory.

pic5

You don’t need to worry about this file.  However, you’ll notice the repository settings won’t work very well.  I’m not going to bother with them for now.  Next, we’ll create a script to setup our project:

pic6

The first line installs gulp and the next two lines install two modules we’ll use in our first test.  When you run the script you’ll see a bunch of status information as the components are installed.  After it’s all done the directory will look like this:

pic7

You can explore node_modules to see what has been installed.

For our testing we’ll create additional directories and some test files:

pic8

All I want is some stuff to work with and that is what testSetup.ps1 provides.  Now we will create gulpfile.js:

pic9

Here is what is going on:

Lines 6, 7, 8 tell node what packages we require and provide a short cut to access the modules.

Line 10 creates an object call paths that we will store values in.

Lines 14, 15, 16, and 17 store the paths we will work with in the paths object.

Line 19 creates a task called “clean:dest” that uses the gulp-clean module to delete files.

Line 24 creates a task called “min:txt” that uses the gulp-concat module to concatenate all the files together.

Line 30 and 31 setup tasks that can call multiple tasks.  In this case if I had also created a min:images task I would make line 30 look like:

pic10

This would cause gulp to first call the task “min:txt” and then “min:images”.

The final task “default” first calls “clean” and then calls “min”.  Clean deletes all the files in destination and min creates the new file and puts it in destination.

Now that we have our gulp file we can run it right?  In a Visual Studio project, we’d be good to go.   If you look inside node_modules in the gulp folder you’ll see “gulp.bat” and if you look at it in notepad you’ll see what is going on.  To do the same in PowerShell, just create gulp.ps1 in the project directory.

pic11

At the command line if we run .\gulp.ps1 we’ll get the following results.

pic12

Looking in our destination folder we should see bigText.txt and it should have the contents of all the other files we put in our source directory and sub-directories.

Where to now?

At this point we really haven’t done that much with Gulp, but we have a good start to play with it and learn how to work with it.  My next step will be to review the modules that are available and learn to work with them.

My goal is always to automate routine tasks and find ways to stream line workflows.  I can see Gulp fitting in with PowerShell to help me accomplish those goals.

Stuff to look at 2017-07-25

Getting started with Power Shell

I make heavy use of PowerShell in a lot of difference scenarios.  It allows me to automate work and free me so I can focus on other things.  There are other scripting languages to learn (BASH for example) but if you’re running Windows you really should know PowerShell:

Getting Started with Windows PowerShell

This is a good starting point that leads to more stuff.  Just start digging.  Your machine should have powershell so you can play along.

ASP.NET Core

I come from a Microsoft Platform background so naturally this is what I use.  The advantage that .NET Core offers me is that it runs nearly anywhere today so I can build applications for to run on iPhones, Android, Linux, iOS or even Windows.  ASP.NET is how web sites are build, ASP.NET Core is the way to build web-sites that can be hosted wherever .NET core is running.  In my case I’ll be targeting linux containers running various middleware packages (Apache and MySQL for example) that will be hosted in Docker environments (like OpenShift, Azure, AWS, etc).  In addition to learning the ASP.NET stuff, you’ll also want to learn C#.  It’s both easier and harder than JavaScript.  Here are a few starting points:

C#

Getting Started with C#

C# Fundamentals for Absolute Beginners

ASP.NET Core

Introduction to ASP.NET Core

 You should also check out the Microsoft Virtual Academy which provides a lot of free training material.  There is also Microsoft’s Channel 9 which is well worth you time.

Storage and Databases

Data has to be stored somewhere.  There are a lot of choices available.  To get things kicked off though I’m going to recommend you start off learning to use MySQL (google it yourself).  Once you’re able to work that things like MongoDB and Microsoft SQL Server will be worthwhile to look at.  You’ll also want to learn to use cloud storage like Azure Table and Blob storage.

You’ll end up learning SQL (Structured Query Language.  You can pronounce it as sequel or squeal depending upon your mood).

The Cloud

You are going to want to know how to work with cloud offerings like Microsoft’ Azure and Amazons Web Services (AWS).  Both are huge offerings with tons of features.  Additionally, you’ll want to know about Docker and OpenShift because containers are becoming a very important feature of cloud ecosystems.  You can get trail accounts for all of them and all of them offer extensive documentation and tutorials to help get you going:

Microsoft Azure

AWS

OpenShift

Regarding free trials – don’t be abusive but I recommend creating an alternate email account (use outlook.com or gmail) instead of your usual email address.  If you need more time just create a new one.

DevOps

DevOps is all the rage, but in a lot of ways it is not that big a deal.  In my opinion, it is just the natural evolution of what we do.  The goal is to shorten the time from having an idea to putting it in production.  That really is all it is about.  You’ll see terms like CI/CD thrown around (Constant Integration & Constant Delivery) but again: it’s not a big deal.  It’s just learning to use the tools to full ability.  However, there are a lot of tools out there.  Typically, I like to choose just 1 tool and get very good at it.  In many cases though you’ll find that there are many tools that do that same thing slightly differently and in some cases, you HAVE to use one particular tool because another tool you are using depends upon it.  For instance: OpenShift depends upon Ansible for its deployment and maintenance.  Ansible is like Chef the tool I’d actually recommend you learn to use.  As a result: you will end up learning several different tools.  This is sort of a catch all.  But my top tools are:

You’re also going to want to learn Linux in addition to Windows.  The entire ecosystem is changing and being just a Windows developer or just  Linux developer doesn’t really work anymore.  You want to be able to target anything and handle any situation.

So this should keep you busy for the day.  I’ll have more stuff tomorrow.  Enjoy

Better living through PowerShell – Archive old pictures

I’m lazy, I hate having to do the same thing twice.  It’s boring.  If there is something that needs to be done regularly I want it automated.  That’s why I became a software developer.

Naturally: the cobbler’s kids have no shoes.  I’m kept so busy making things better for my clients that I forget to take time to make things better for myself.  Time to start changing that

In this case, I have a fairly simple problem: I take lots of pictures on my cell phone.  My cell phone uploads the pictures to my OneDrive.  I use OneDrive on all my machines.  My Surface has a 256Gb SSD in it.  There are about 127Gb of pictures on my OneDrive.  See the issue?

My main workstation has a 4Tb external hard drive where I kept ‘stuff’ and it has a media folder where old pictures and videos are stored.

The requirements seem easy:

  • all pictures in my Camera Roll folder on OneDrive need to be moved to my Camera Roll folder in my media archive.
  • I also want video files (MP4) that are more than 5 days old to move as well.
  • I’d also like to keep a log so I can tell that this is happening.

The way I chose to implement this (the title should get it away) was to write a very simple PowerShell script and then schedule it using Windows Scheduler.

Here is the script:

script

While this is easy, I had to look up this stuff so I don’t feel bad explaining it in detail.  I’ll break it down step by step.

Before I get in to it though I want to point out that you can type each of the commands above in to the PowerShell command line to see what they do.  You can even change things around to see what happens.  Or you can load the script in to the PowerShell ISE and step through it with a debugger.

The first part of the script just counts how many files are going to be moved.

The very first line (after the comment) is just giving me a short cut to where I’m going to write my logging information.

Next get-childitem gets all the JPEGs in my camera roll.  Get-childitem is also aliased as the old DOS “DIR” command or UNIX ‘ls’ command if you don’t want to type all of those letter.   The list of files is passed on to where-object using a pipe command (the ‘|’ thing).

The where-object filters out pictures that are less than 30 days old and then passes them, or pipes them on to measure-object.  Measure-object counts and measures stuff.  In this case we just want the count.  Notice that I have all the commands wrapped inside of parentheses?  This allows us to get just the count output from the measure-object command and assign it to our $jpegCount variable.

line1

The exact same process is used to get a count of video files in my camera roll.  The ony differences are that we are looking mp4 files and we only want files more than 5 days old.

Next I make a custom date string by getting today’s date with get-date and passing it to ToString with a custom format string.  I like formatting dates from greatest precedence to least precedence.  It makes more sense.  You don’t write Fifty five dollars two hundred do you?  Then why do we write dates as April 27, 2017?

With our nicely formatted date and file counts we now update the log file with the information about how many files are about to be moved.  Add-Content appends whatever you provide to the file you specify.

line 2

With the log file updated we’re now ready to move the files around.  Moving the pictures looks just like getting the count.  The differences are that we are not assigning the results to a variable so we don’t need the parentheses and we pass the filtered list of files to move-item instead of measure-object.

line 3

Moving the videos is the same process with the differences as explained before.

We now have our script that does the work.  The last step is that we need to schedule the script to be run once a day.  To do this run Task Scheduler as the administrator and create a new task.  Here is how I do it:

  1. Window+Q to bring up Cortana and search for “Task Scheduler”

cortana

  1. Right click on “Task Scheduler” and choose “Run as Administrator”
  2. In Task Scheduler click on “Create Basic Task”

create basic task

  1. Give it a name and click “Next”

cbt1

  1. Set the trigger to daily and click “Next”

cbt2

  1. Select a time for the job to run. I like to do things at 1am.  Click “Next”

cbt3

  1. Select “Start a program” and click “Next”

cbt4

  1. Now we enter our actual program. In this case you can just type the entire command line in the main box.  The command you want to type is:

PowerShell -noninteractive -file C:\Users\your user name\Documents\WindowsPowerShell\MoveOneDrivePictures.ps1

With obvious adjustments.  When you’re done click “Next”

cbt5

  1. You’ll be presented with a confirmation message b/c Task Scheduler is going to split things up for you. Just say “Yes”

cbt6

  1. You’ll be presented with a summary of what you want scheduled. If you’re happy just click “Finish” and you’re done.

cbt7

You can test our your job by right clicking on it and choosing “Run”.  Just go see if the log file was created.  If it was your job ran.  Otherwise you have some debugging to do.

Overall this is not a horribly complex script or task, but it opens the door to other things.  We’ll get in to those things soon.

Rescue Support: Problem Solving

Getting started learning a new piece of tech can occasionally be frustrating.  In my case I ran in to what I suspect is common problem with open source software.  Namely: support and what do you do when there isn’t any?

To set the stage I’m learning AngularJS 1.  The reason I’m staying on 1 and not 2 might be a topic for another day.  I’m using Plural Sight to help me learn and a lot of the videos I’m watching are by John Papa.  He has a pretty cool framework called Angular HotTowel that is easy to add to a Visual Studio project.  Here is where our trouble starts.

The trouble starts when I tried to add the angular UI grid to my project.  It blows up because HotTowel is using an old release of AngularJS that was not compatible with Angular UI Grid.  Inside Visual Studio’s package manager all the libraries that HotTowel used had newer versions available.  But when you update to them HotTowel’s navigation breaks.

This is where software development problem solving takes over.  Running in the debugger I can trace through the code and see what is happening but it’s not clear why it is not working.  Reading the Angular library code is very time consuming and I just want to get on with things.  Surely somebody else has experienced this problem already so off to the search engines.

Googling the problem “HotTowel navigation broken” reveals nothings.  Lots of variations and different combinations made no difference.  the next step is to ask.  The obvious place to me was the HotTowel project on GitHub.  Thing brings up another piece of software development problem solving: clearly explaining what the problem is and then providing the steps to reproduce the problem.

Explaining and problem and showing people how to reproduce is can be difficult and time consuming.  However, I’ve usually found the effort worthwhile and in many cases, it leads to the answer before you even ask somebody else.  To get started I needed to isolate the problem to determine if it was an Angular problem or a HotTowel problem.  This lead to me reading a lot of material on how Angular does navigation.  It also put a spot light on the problem for me.

I started off with this article by Viral Patel “AngularJS Routing and Views Tutorial with Example”.  I used his sample code and had it up and working.  Viral is even so nice as to provide a plnkr.co environment to play around.  His example is nearly identical to what HotTowel is doing, even the early version of AngularJS.  If you change line 41 so that it refers to version 1.6.2 of AngularJS we reproduce our problem.   We’re making progress.

Something in Angular’s routing changed.  To try and understand what is happening I went off to Angular’s site (probably a better place to start in the first place, but it might not have revealed the problem so quickly).  In their tutorial (which is excellent by the way) there is a section on routing and views.  In the section on Configuring a Module we start to see what changed:

20170323 pic1

What is this hashPrefix nonsense?  Where did that come from, what does it do and is it causing my trouble?  Easy enough to find out.

Returning to the plnkr we just broke we can switch to the app.js file and change it to be like the one in the tutorial and see what happens.  We need to add ngRoute to our application and then add $locationProvider to the config function and finally setup the hashPrefix.  Easy:

20170323 pic2

Unfortunately, that doesn’t appear to have fixed the problem.  Reading in more details in the Angular tutorial we will notice a little green box with this nugget of wisdom:

20170323 pic4

Our html links for adding an order or showing an order do not have bangs in them (‘bang’ is short hand for the exclamation mark, ‘splat’ is for the asterisk ‘*”, I forget the other Unix cool guy slang.).  Return to the plnkr and see if just adding a bang to the path fixes our problem.

In plnkr return to the index.html file and add bangs after the hash marks:

20170323 pic5

Unfortunately, things still don’t work.  But after playing around a little more I removed the locationProvider from the config method and things started working.  So the config method ends up looking like this:

20170323 pic6

At this point things work.  With that in mind I return to my HotTowel application and make a few changes.  HotTowel already includes ngRoute, so all I have to do is change how SideBar.html creates the links:

20170323 pic7

Running the app and retesting shows that the simple addition of a bang (‘!’) to our links fixes the problem.  Talk about annoying.

For a person who has been working with Angular for a long time this might not seem like a big deal.  The reason I’m taking the time to walk through this process is to show how methodically breaking a problem down will help you solve it.  As you gain experience with a technology you’ll still encounter problems that in hind sight are trivial but at that moment bring you to a standstill.  The key is to isolate the problem in a reproduction.

Had my reproduction of the problem not led me straight to the solution I would have used the reproduction to ask the community at large what was going on.  Keep in mind that everybody is busy so the more succinct you make your question and reproduction the better your odds are for getting help.

I recommend asking for help as quickly as possible.  In case like posting to StackOverflow or the MSDN forums you might not get an answer for days, if at all.  So posting a question while you continue your own work is a good strategy.  In some cases, you’ll arrive at the answer before anybody else answers the question for you.  In others, somebody will point you in the right direction.  As a final not on asking for help: don’t forget to ask your own team.  Whether its work or your social circle: maybe somebody there already has the answer or can help you figure it out.

Custom .NET Configuration Sections

In September of 2015 I wrote a more extensive blog post that covered this material.  However, it covered a lot of other material and buried what I’m discussing here.  If you want more information on how to make your configurations work with custom build configuration read “Configuration Notes

You are writing a cool new application that needs some of its settings to be stored in a configuration file.  Happily .NET has always provided a nice configuration subsystem.  In general everybody knows how to use it.

What I’m going to point our here is that you don’t need to use appSettings for your configuration stuff.  In my experience appSettings is a horrible place to put your configuration information because it becomes crowded with stuff.

The alternative requires no code and in fact is still appSettings but allows you to organize your settings to keep different components separated from each other.   In my current project I need to store the names of tables in Azure Table Storage being used by the service.  Here is what my configuration file looks like:

custom .net config sections app.config file

The secret sauce is in the type tag for our RescueServices section.  The type, NameValueSectionHandler is the same type used for appSettings.  You can verify this by finding the machine.config file and checking out how the sections are defined.

In order to access your configuration information you just need to write the following lines of code:

custom .net config sections code snippet

This is the only downside to not being in appSettings: configuration manager doesn’t provide a short cut.  Instead you first have to ask for your section and tell it the type to use.  For us we just use a NameValueCollection (same as appSettings) and we’re good to go.  Once you have the collection you can go to town.

A truly lazy developer would create a base class that handles all of this so that all you must do is provide the name of the configuration section and it handles the rest.  That is left as an exercise for the reader.

Better Question

Sorry to say but people do not seem to ask good questions.  In fact I find that people don’t seem to ask questions at all.  I notice people making a lot of assumptions without ever reflecting on whether they got their assumption right.

When people do ask questions, they seem to ask the wrong questions.  The usual question coming out of people is essentially “why?”.

  • Why did you write the component like that?
  • Why aren’t there any unit tests?
  • Why is your project behind schedule, over budget and full of bugs?”

A “Why” questions elicits a belief answer.  When they do setup a “why” question that gets a non-belief based answer they have injected the question full of their own beliefs to get that answer.  A better and easier way to ask questions is to use the other W’s instead.  Who, what, when, where, whow (the W is silent).

  • How many unit tests are there?
  • What is this testing?
  • When are the tests being run?
  • Where are the tests run?
  • Who writes the tests?
  • How do you decide what gets tested?

Part of the reason I avoid belief based questions is to avoid distractions.  Generally, if I’m involved there is a goal that I am trying to achieve (say get a project back on schedule) and I don’t have time to philosophical debates about anything.  I save those discussions for after hours with yummy beverages and delicious food.

Stepping back and asking questions is an important skill.  Learning to ask useful questions is an even more important skill to develop.  For instance, “What would you need to get this million-line project to 80% code coverage in unit testing in 10 days?” is sometimes a good question to ask.  It will start a conversation, probably a long one, that will get the team thinking outside their normal patterns.  In many cases this is all that is needed.

Using questions skillfully can also help develop and deepen relationships with people.  This can be useful helping bring a team together.

Rescue Support – Technologies

As I set out to build rescue support I have specific technologies in mind to use. It should be clear that I reserve the right to grab whatever technology catches my fancy along the way.   Getting started, the menu will include the following technologies:

  • XAMARIN Forms
  • HTML5/CSS3/JS
  • AngularJS
  • Breeze
  • ASP.NET Web API & MVC
  • Azure
  • Docker Containers
  • Microservices

Part of the issue developers always run in to is focusing on one set of technologies means that they are missing out on others. I’m not really going to pay attention to Node.JS, AWS, and a bazillion other ‘new’ developments. There are also a lot of technologies I’ve not even heard of yet but will probably sound really cool later.

I’m also not listing everything here. For instance, Bower and some of the ‘new’ tools that have appeared on the scene aren’t listed but I’ll probably talk about them at some point. There are also the frameworks that are built on top of these tools. For instance, I’m going to use HotTowel as the starting point for my client. I also make use of Unity, Enterprise Logging and several libraries on the server side to make my life easier. I plan to talk about all of them along the way.

The wonderful thing about being a developer is that no matter what time it is there is always something new to learn. The scary thing about being a developer is getting left behind, waking up one day and finding out that nobody cares that you can seriously rock some IUnknown with your mad STL skills.