sverrirs.blog

VSIX Extension Gallery for Visual Studio

Sun, 21 May 2017 08:00:01 +0200

In a [previous article](/2017/05/vs-itemtemplates-wizards-and-vsix.html) I discussed how to create Item and Project templates and bundle them into their own VSIX installers. However, now that you have your VSIX installer the question becomes how do you distribute it to all your coworkers? It would be great if they could use the _Extensions and Updates_ manager that is already built into Visual Studio. That handles installing, uninstalling, searching and auto-updating extensions. Pretty neat! However the project you have is not suitable to have in a public VSIX repository like the [Visual Studio Marketplace](https://marketplace.visualstudio.com/vs).

The public marketplace is no place for internal company extensions unfortunately.

You need to host your own private marketplace. **But how?** TL;DR; I built one # Private Extension Galleries Microsoft has [addressed this issue](https://blogs.msdn.microsoft.com/visualstudio/2011/10/03/private-extension-galleries-for-the-enterprise/) but only through documentaton. There are no concrete implementations available from Microsoft nor any others on how to host and serve these files. Neither is there a simple way to leverage the advanced features the Extension Manager provides (such as search, ratings and download counts). ## So what is available? Unfortunately the commercial offerings are incredibly limited. The main one [myget.org](https://www.myget.org) is purely online and regretfully not free. The popular [Nexus Repository](https://www.sonatype.com/nexus-repository-oss) by Sonatype dropped support for VSIX files in their latest version (v3). There are some [half-automated](https://github.com/garrettpauls/VSGallery.AtomGenerator) solutions out there, others [very manual](https://www.codeproject.com/Articles/881890/Your-Private-Extension-Gallery). The worst thing about most of the automatic offerings is that they require being run on an [existing webserver](http://blog.ehn.nu/2012/11/using-private-extension-galleries-in-visual-studio-2012/) (IIS, Apache, Ngix, etc) and require a relational database system to store data. So there really is no freely available out-of-the-box solution available. **Until now...** # Introducing vsgallery

The VS-Gallery running inside of Visual Studio's Extension Manager

With the current rise in popularity of [_the Microservice_](https://en.wikipedia.org/wiki/Microservices) I found it really disappointing that a simple click-to-run solution wasn't available to run a private Visual Studio Marketplace. I wanted something simple and self-contained that could be run without installing and configuring multiple other systems (such as a webserver and a database system). ## The VS Gallery solution Before I bore you with more text, go ahead and test [_vsgallery_](https://github.com/sverrirs/vsgallery) out. Just download the latest release and run the executable file. It is really that super simple. I promise! Download vsgallery **vsgallery** is a single executable file which acts as a complete self hosted extension gallery for Visual Studio 2010 and newer. It really is ultra simple to configure and run. You are up and running in a few seconds. All for the low low price of **FREE**! The whole system runs as a single self-contained executable and uses no database. All files and data are stored on the local file system which makes maintenance and backup super simple.

## Features * Fully featured Extension Gallery ready to use in Microsoft Visual Studio. * Counts downloads of extensions * Displays star ratings, release notes and links to project home pages * Offers a simple to use REST API to submit ratings and upload new VSIX packages * Atom and JSON feeds for available packages * It's FREE! # How to install into Visual Studio In Visual Studio ``` Tools > Options > Environment > Extensions and Updates ``` Add a new entry and copy in the URL of the main Microservice Atom Feed. > By default the URL is `http://YOUR_SERVER:5100/feeds/atom.xml`

Please consult [this MSDN document](https://msdn.microsoft.com/en-us/library/hh266746.aspx) for any further details and alternative options on how to install a Private Extension Gallery in Visual Studio. # How it works The microservice is configured via the `config.ini` file that sits in the same folder as the main executable. The `.vsix` files, along with their download counts and ratings data are stored in a subfolder of the main service executable `VsixStorage/` (this subfolder is configurable). This makes taking backups and moving the service between machines super easy as the root folder contains the entire Microservice state and data. ``` root-folder |--vsgallery.exe |--config.ini |--VsixStorage |--atom.xml |--First.vsix |--Second.vsix |--AndSoForth.vsix ``` Latest release # The vsgallery API The Microservice comes with a rich HTTP based API. You can plug the data and its functionality directly into your development portal or company intranet with minimal web programming. Even direct integration into your continuous integration platforms and communication pipelines such as #slack are possible. > The `vsix_id` required by many of the endpoints can be obtained by reading the `id` field in the feed endpoints. ### [GET] /feeds/atom.xml This is the main entry point for the VSIX feed and serves up the Syndicate-Feed compatible Atom file containing all available extensions on the server. **This is the URL endpoint that should be used in Visual Studio.** See [How to install into Visual Studio](#how-to-install) for more information. ### [GET] /api/ratings/{vsix_id} Retrieves the rating value and vote count for a particular VSIX package by its ID. ``` curl -X GET http://VSGALLERY_SERVER:5100/api/ratings/VSIX_ID ``` The return type is the following JSON ``` { "rating": 4.3, "count": 19 } ``` ### [POST/PUT] /api/ratings/{vsix_id} Submitting rating values for a particular VSIX package by its ID. The post payload should be just raw string and contain a single floating point value in the range between [0.0, 5.0]. The example below will post a rating of `3.5` stars to VSIX package with the id `VSIX_ID` ``` curl -X POST -H "Content-Type: text/plain" --data "3.5" http://VSGALLERY_SERVER:5100/api/ratings/VSIX_ID ``` ### [GET] /api/json JSON feed for the entire package catalog. Same data that is being fed through the atom feed but just in a handier JSON format. ### [POST/PUT] /api/upload This endpoint accepts form-data uploads of one or more .vsix files to the hosting service. The example below will upload the file `my.vsix` to the gallery server and propose a new name for it `renamed.vsix` (you can omit the filename param to use the original name) ``` curl -X POST --form "file=@my.vsix;filename=renamed.vsix" http://VSGALLERY_SERVER:5100/api/upload ``` To upload multiple files simply add more form elements. The example below uploads two VSIX files at the same time. ``` curl -X POST --form "file1=@my.vsix" --form "file1=@your.vsix" http://VSGALLERY_SERVER:5100/api/upload ``` # Closing So if you're searching for a simple solution for your internal, low traffic, extension gallery then please consider **vsgallery**. If you do try it out, please leave me feedback in the comments below. Peace! vsgallery

Visual Studio Item Templates, Custom Wizards and VSIX installers

Sun, 21 May 2017 08:00:00 +0200

**News flash** You can create your own item and project templates for use with Visual Studio! Well I guess this is not really so big news if you've used the IDE for a while. Templates for files housing source code, resource configurations, configuration files and many other things can in many cases quickly out weigh any cost that it entails creating one. Especially if you're working on a large team or that your code base requires a fair amount of tedious but important boilercode for new source files (even things like header comments can be a huge timesaver).

Add New Item dialog">

A custom item template in the Add New Item dialog

So we all know that these things can be created. What is surprising is really how easy it is to create them. There are numerous sources all over the internet that discuss how to create both simple item templates and larger more complex project templates. Here are a few good ones: [#1](https://blogs.msdn.microsoft.com/vsx/2014/06/10/creating-a-vsix-deployable-project-or-item-template-with-custom-wizard-support/), [#2](https://msdn.microsoft.com/en-us/library/ms185301.aspx), [#3](https://blogs.msdn.microsoft.com/akirsman/2012/10/11/creating-visual-studio-templates/), [#4](https://blogs.endjin.com/2014/07/a-step-by-step-guide-to-developing-visual-studio-item-templates-using-sidewaffle/), [#5](https://msdn.microsoft.com/en-us/library/ms247064.aspx), [#6](https://msdn.microsoft.com/en-us/library/eehb4faa.aspx), [#7](https://msdn.microsoft.com/en-us/library/xkh1wxd8.aspx) but really they are too numerous to list here. So for this article I will not go into the details of the actual creation as it is fairly mundane. What is however hard to comeby is a complete actual working example of one. This is what I have for you! See the code In addition to the complete example solution I also want to focus on the nitty-gritty issues that crop up when composing these types of projects that very few sources seem to discuss in any detail. ## The project structure This is an overview of the current example solution I am offering. Just having a reference to what goes where and what references are useful where can be very helpful, so here you go.

The complete file structure of the solution

## Choose correct project types After installing the Visual Studio Extension SDK it is very important that you choose the correct project types. These are all found under the `Visual C# > Extensibility` category.

Yes even though you're creating a C++, F# or a template for what ever other language, the project housing the templates and relevant code will always be in C#. ## Ensure you use the correct item tags The `.vstemplate` file that contains the configuration for your Item or Project template must be configured as a `` item in the template `.csproj` file, [Example](https://github.com/sverrirs/vstemplates/blob/master/VSItemTemplate/VSItemTeamplateFiles/VSItemTeamplateFiles.csproj#L58). This setting is controlled via the _Build Action_ for the .vstemplate files. Ensure that this is correctly set, otherwise your files will not be built and packaged correctly.

VSTemplate for all .vstemplate files">

Choose build action VSTemplate for all .vstemplate files

When you create a new project from scratch this value is correctly set for the existing file. However when you add your own files in there this is a very common oversight to make. ## Keep your Wizard code in the VSIX installer project This is by far the simples approach and requires only two projects. If you really really need a third project for the wizard code then this is not a huge problem but can be a bit of a headache to setup in the VSIX installer configuration (hint: use dependancies). > The Wizard code cannot belong to the Template project as the output of this project will not be a dll but a zip file containing your templates. ## Finding the correct dlls for the Custom Wizard The two external dll includes you need for the custom wizard logic are the `Microsoft.VisualStudio.TemplateWizardInterface.dll` and `envdte.dll`. It is very likely that there are multiple versions of these dlls floating around on your computer. > Choose the ones that were shipped with the Visual Studio edition you're building your project in and/or the version of the IDE that you're specifically targetting. For me this means including these versions: ``` C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.TemplateWizardInterface.dll C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\envdte.dll ``` ## You must sign the Wizard assembly For Visual Studio to even consider loading the dll containing your custom wizard implementation you must sign the assembly containing that code. It is not a biggie and just means you need to specify the fully qualified name of the assembly in the `.vstemplate` file. I really love [this trick](https://blogs.msdn.microsoft.com/miah/2008/02/19/visual-studio-tip-get-public-key-token-for-a-strong-named-assembly/) to set up an _External Tool_ to extract the PublicKeyToken for the assembly. 1. `Tools > External Tools` 2. Add new one and use the following config 3.

``` Command: path/to/your/sn.exe Arguments: -T $(TargetPath) Check: Use Output window ``` 4. In the Solution Explorer: Select the assembly and run your newly created tool. ## When testing your extension Visual Studio can get confused A common error to crop up is the _"GetDeploymentPathFromVsixManifest" task failed unexpectedly_. This error occurs for no apparent reason, usually after you've been able to build the solution quite a number of times without errors. This error is due to the existence of certain folders and files under your $APPDATA directories that relate to running the Experimental versions of Visual Studio (which is done when you debug your extension). The example solution includes a bat file `VSItemTemplateVISX\prebuild_clean.bat` that will remove all these unnecessary files for you in-case you encounter this error. _You will have to shut down your Visual Studio instance before running the command_ ### The contents of the batch file The batch file simply attempts to delete the *_Exp folders that were created for your Experimental instances from your local AppData caches. The entire bat file contents are below for completeness. ``` set VS_PATH=Microsoft\VisualStudio REM First check Local/AppData cd %LOCALAPPDATA%\%VS_PATH% for /f %%G in ('dir /b "*Exp"') do ( echo Found %%G rmdir /S /Q %%G ) REM Second check the main AppData path cd %APPDATA%\%VS_PATH% for /f %%G in ('dir /b "*Exp"') do ( echo Found %%G rmdir /S /Q %%G ) ``` ### Details about the problem The actual problem lies in the fact that Visual Studio locks the following three files ``` privateregistry.bin privateregistry.bin.LOG1 privateregistry.bin.LOG2 ``` You can get away with just deleting them but removing the entire _Exp folder does no harm and potentially avoids other issues if you leave out of date Experimental files in there. ## Conclusion Well that is it. The example solution should speak for itself and the points above highlight the biggest pains that I had while attempting to create my first template installer. Hope this is of use and as always drop me a line in the comments if you have something to add or any questions. See the code

Mastering cell validation lists in Excel

Fri, 24 Mar 2017 08:00:00 +0100

> "So here is the thing, we want to have a list of values that the user can choose from in the cells of our sheet" > __-Your boss__ Dropdown lists of the sort shown below are very common and can be hugely beneficial for users when filling out Excel sheets. They are quite easy to create and offer basic data validation so that the user can't accidentally type in the wrong value.

Lists with values can be very useful when creating data-entry sheets

**TL;DR; If you just want to jump to the solution then feel free to download the demo sheet straight away :)** Demo File But yeah, back to the data validation lists. You can create one very easily via the ribbon through `Data` >> `Data Validation`

So easy...

_Super!_ > "So yeah, this is great. But we want the list of values to be editable by the user." > __-Your boss__ Wait. That's easy! I can use ranges in these cell lists. So I'll just have one sheet that acts like a "Configuration" sheet and have some column in that sheet hold all the values to show in the list.

A list of colour names, so easy to add new ones at the end... so easy

Then I will just have the list pull data from a big chunk of the range so that the user can add items to the end of the list indefinately.

Oh my, still so easy...

## Too many blanks Unfortunately using an unbounded range for the list items causes a rather annoying bug. In the dropdown lists there will be a large section of completely empty items at the end. Even through you tick the _Ignore blank_ option.

When using ranges a bunch of empty values appear at the end of the list. Annoying!

This is made even worse by the fact that the list always automatically scrolls down to the first blank item when entering values into an empty cell. There are some solutions to this. Using the [`OFFSET`](https://support.microsoft.com/en-us/help/324991/how-to-use-the-offset-function-in-excel) function to limit the number of items in the list box is one of the fastest and simplest.

The OFFSET function can be used to help. But requires a lot of extra cells

However to use the `OFFSET` function you need to know the total number of items to trim to, which requires you to use intermediate cells to count non-blank items. In the image above the green cells in column `C` count up when the cell in column `D` has a value. Then the cell orange cell in column `E` just takes the `=MAX(C:C)` range. This number you can use as the limit to the OFFSET function straight in the data validation box: ``` vb =OFFSET(Configuration!D2:D4000,0,0,Configuration!$E$3) ```

Data Validation using the OFFSET function

For some this is an acceptable solution. But for most this is just too brittle as the user might accidentally delete values in either column C or E and break the entire thing. [Read more about the OFFSET function on MSDN](https://support.microsoft.com/en-us/help/324991/how-to-use-the-offset-function-in-excel). ## Too many items Alright, no worries, you can fix this. How about we use VBA to construct a simple string list of items and insert it directly into the cell validation list? Something like this: ``` vb Public Function ApplyConfigLookupToRange( applyToCells As Range, lookupValuesRange As Range) Dim ddl As String Dim Value As Variant For Each Value In lookupValuesRange If Value = "" Or IsEmpty(Value) Then 'As soon as there is a blank entry in 'the list then exit, list must be contiguous Exit For End If 'Note: Regardless of the language/region set for Excel 'this feature always uses comma (,) as a separator ddl = ddl & Value & "," Next Value 'Trim the last comma from the end of the list If Len(ddl) > 0 Then ddl = Mid(ddl, 1, Len(ddl) - 1) End If 'If there are no items to add to the list 'then just delete everything from the range If Len(ddl) < 1 Then With applyToCells.Validation .Delete End With Else With applyToCells.Validation .Delete .Add Type:=xlValidateList, _ AlertStyle:=xlValidAlertStop, _ Operator:=xlBetween, _ Formula1:=ddl .IgnoreBlank = True .InCellDropdown = True End With End If End Function ``` Then we can simply call this function when we deactivate the Configuration sheet and apply it to our entire range. Something like so: ``` vb Private Sub Worksheet_Deactivate() Call ApplyConfigLookupToRange( Sheet1.Range("$A$1:$A$4000"), ' All cells that should have the list dropdown "Configuration!$D$2:$D$4000")) ' List of colours End Sub ``` Yes, this will work! And it does, for a while... > Note: When specifying ranges like in the `Worksheet_Deactivate` function above. Always use the fully locked formula with `$` signs for both columns and rows. ## Error unreadable content in workbook For a while the VBA solution works. But then suddenly your users start complaining about a strange error that keeps coming up when they open their sheet in the morning. They constantly have to have Excel "recover it" and when they do it seems that some content gets removed during repair. They're worried!

Super scary errors, what did you remove! Tell me... WHAT!!

You can't find anything wrong with your code. But the error is easily reproducible. There is something wrong. Everytime the sheet is closed and reopened the error appears. You check, but the users **really** haven't deleted anything critical or messed with the formulas. They've just been adding more colours to the list as they were instructed to do and was a major selling point for them to use your Workbook. Then you find it. _Bam!_ There is an undocumented max-limit for string lists! They cannot be longer than 256 characters! _groan_ So you cannot use your VBA solution after all. What to do? ## Hybrids to the rescue The real solution involves combining both of the approaches above in a way that circumvents the string limit of 256 characters while still keeping the sheet robust enough to handle users deleting rows willy-nilly without stuff breaking. _Tired? Want to jump to the solution?... Go ahead :)_ Demo File We modify our VBA code to produce an [OFFSET](https://support.microsoft.com/en-us/help/324991/how-to-use-the-offset-function-in-excel) function instead of a string. Then we make this function operate on the value range and places it in the data validation logic using VBA! Like so: ``` vb Public Function ApplyConfigLookupToRange( applyToCells As Range, lookupValuesRangeName As String) ' First we must count the values in the range Dim numItems As Long ' Now evaluate the range and loop through it ' counting all of the consecutive values Dim lookupValuesRange As Range Set lookupValuesRange = Range(lookupValuesRangeName) For Each Value In lookupValuesRange If Value = "" Or IsEmpty(Value) Then Exit For End If numItems = numItems + 1 Next Value ' Stop executing if nothing in the range If (numItems < 1) Then With applyToCells.Validation .Delete End With End If Dim formulaTemplate As String formulaTemplate = "=OFFSET(" + lookupValuesRangeName + _ ",0,0," + Str(numItems) + ")" ' Now create the drop-down validation boxes ' for the entire cell range that was passed in With applyToCells.Validation .Delete .Add Type:=xlValidateList, AlertStyle:=xlValidAlertStop, _ Operator:=xlBetween, Formula1:=formulaTemplate .IgnoreBlank = True .InCellDropdown = True End With End Function ``` The way we call the function is still the same ``` vb Private Sub Worksheet_Deactivate() Call ApplyConfigLookupToRange( ' Cells that should have the list dropdown Sheet1.Range("$A$1:$A$4000"), ' List of all available colours "Configuration!$D$2:$D$4000")) End Sub ``` That's all there is too it. You circumvent the string limitation, keep the necessary elements for OFFSET out of the way of the user and have a list of values that the user can expand practically indefinitely. _fanfare!_ ## Note on Named Ranges I want to recommend the use of [named ranges](https://msdn.microsoft.com/en-us/library/office/gg192736.aspx) instead of the COL:ROW + $ pattern. Named ranges are preferable as their extent can be modified without going behind the scenes to manipulate the raw VBA code. They also do not need any special `$` handling to avoid a "shift" when generating the OFFSET functions and applying them to the cell ranges. Managing the named ranges can then be done by non-programmers after the sheet is delivered to its users. In the example below I have replaced the hard-coded COL:ROW patterns with named ranges: ``` vb Private Sub Worksheet_Deactivate() Call ApplyConfigLookupToRange( Sheet1.Range("DataEntry_Colour_Cells"), "List_Colour_Names") End Sub ``` [Read more about ranges on MSDN](https://msdn.microsoft.com/en-us/library/office/gg192736.aspx). ## Note on calling the code > And where should I then call this code? Ideally I think this code should be called directly on `Worksheet_Deactivate` when leaving the configuration sheet. That way the correct cells are always kept up to date after the user changes the list contents. Now for the actual working demo, check it out! Demo File Happy coding!

Extracting values from HTML forms as JSON the easy way

Sat, 18 Mar 2017 08:00:00 +0100

Extracting values from HTML form elements in webpages has historically been and still is a messy business when done client-side. Usually the process entails a rather cumbersome hand-written javascript function that acesses each DOM form element and extracts its value into a variable for later use. Many of the modern MVVM frameworks solve this by using view-model classes that are data-bound to the elements. However for the vast majority of us that don't have this luxury then we're usually stuck with a brittle javascript logic that has intimate knowledge of the HTML DOM structure. This unfortunately makes our code hard to maintain, difficult to test and close to impossible to reuse. ## How about JSON? For a [recent project](https://labs.coruscantconsulting.co.uk/seo/socialmedia/index.php) I was yet again in the place of having to extract a large amount of data out of form elements for client side manipulation. For this project (which uses [handlebars-js](http://handlebarsjs.com/)) I needed to convert the user input into a JSON data structure. I came up with a generic way to very easily extract all user entered data automatically by relying on custom DOM entity attributes. Perhaps an interactive demo explains this best :) Live Demo ## How it works The only thing you need to do for your HTML elements is to decorate them with an additional attribute. By default this attribute is named `serialize-as` but you can use any other name you want. This attribute value will control where in the resulting JSON the field value will be written. ``` html ``` results in ``` js { site_name: "My Awesome Site" } ``` Nesting can be achived by using dots `.` in the serialization attribute (there is no limit on the depth of nesting that can be done). ``` html ``` results in ``` js { site: { name: "My Awesome Site" } } ``` Go ahead, try this out in the Live Demo ## Splitting values The code supports automatic splitting of entered value into JSON arrays based on a delimeter. An additional attribute can be applied, `serialize-separator` that contains the delimiter to split on. This attribute name can also be customized. ``` html ``` results in ``` js { "keywords": [ "software", " services", " seo" ], } ``` ## Producing alternative JSON structure A useful feature when working with templates and fragments of HTML is to be able provide the data in multiple different formats. The jsonify function supports that by allowing you to specify as many alternative serialization attributes as you want for a form element. Consider the HTML below ``` html ``` The data in this input element can be serialized in three different ways depending on what serialization attribute we instruct the function to use. Calling ``` js jsonify() ``` results in the `serialize-as` attribute to be used (the default) and the following json to be created ``` js { website_name: "My Awesome Site" } ``` However ``` js jsonify(null, 'serialize-hb') ``` results in `serialize-hb` to be used which produces ``` js { site: { name: "My Awesome Site" } } ``` and finally ``` js jsonify(null, 'serialize-angular') ``` results in `serialize-angular` to be used which produces ``` js { post: { info:{ meta:{ name: "My Awesome Site" } } } } ``` > This same trick of using different attribute names, also works for the `serialize-separator` attribute. Live Demo ## Optional options The final argument to the function is an option configuration. Here the code can be instructed on what to do if they encounter an empty field. By default a `null` value will be assigned but this can be overwritten by setting `usenull: false`. Also the code will attempt to parse numerical values correctly in the JSON, this can be turned off by setting `parsenumbers:false`. ## The code For those of you who just want the final `js` code. Then you can find the complete code below. > This version requires `jquery` ``` javascript function jsonify(parentEl, attributeName, separatorName, options = {'usenull':true, 'parsenumbers':true}) { parentEl = typeof(parentEl) === typeof undefined || parentEl === null ? document.body : parentEl; attributeName = typeof(attributeName) === typeof undefined || attributeName === null ? "serialize-as" : attributeName; separatorName = typeof(separatorName) === typeof undefined || separatorName === null ? "serialize-separator" : separatorName; var json = {}; $(parentEl).find('[' + attributeName + ']').each(function() { var val = null; if ($(this).is(':checkbox') || $(this).is(':radio')) { var tmpValueAttr = $(this).attr('value'); if (typeof tmpValueAttr !== typeof undefined && tmpValueAttr !== false) { if (!$(this).is(':checked')) return; val = $(this).val(); } else { val = $(this).is(':checked'); } } else { val = $(this).val(); var tmpSepAttrVal = $(this).attr(separatorName); if( tmpSepAttrVal && val ) val = val.split(tmpSepAttrVal); } if( options && options['usenull'] && (typeof val === typeof undefined || val === "" ) ){ val = null; } if( options && options['parsenumbers'] && val && !Array.isArray(val) && typeof(val) !== "boolean" && isFinite(val) && !isNaN(parseFloat(val)) ) { if( val.length > 0 && val[0] != "0" ) val = !isNaN(parseFloat(val)) ? parseFloat(val) : parseInt(val); } var path = $(this).attr(attributeName).split('.'); var path_len = path.length; data = null; $.each(path, function(index, key) { if (data == null) data = json; if (index + 1 === path_len) { data[key] = val; } else { if (!(key in data)) data[key] = {}; data = data[key] } }); }); return json; } ``` I found this approach to be extremely helpful and I hope this will be of use to some of you. Live Demo

Loss of accuracy when parsing Google Maps API data in PHP

Mon, 06 Mar 2017 08:00:00 +0100

I recently came across a rather subtle and nasty bug when using PHP to parse the responses from the [Google Elevation API](https://developers.google.com/maps/documentation/elevation/intro). Example response from Google Elevation API ``` js { "elevation" : 1701.672119140625, "location" : { "lat" : 39.72886619204346, "lng" : -105.1162936849545 }, "resolution" : 4.771975994110107 } ``` The same response data after being parsed by `json_decode` ``` js { "elevation" : 1701.6721191406, "location" : { "lat" : 39.728866192043, "lng" : -105.11629368495 }, "resolution" : 4.7719759941101 } ``` I couldn't figure out why the data I observed when debugging the PHP code did not match up to the data I saw when I called the [Elevation API](https://developers.google.com/maps/documentation/elevation/intro) directly. When cross referencing values from the Elevation API I could never find the corresponding data on the PHP side. However when I searched the other way around I could only ever find partial matches to any of the numerical data. This was especially troubling as I am dealing with high precision GPS data and loss of fractions can sometimes mean a loss of hundreds of meters in precision. I finally tracked the problem to my use of the [`json_decode`](http://php.net/manual/en/function.json-decode.php) function when I was parsing the raw API data. Apparently this function has a known [precision-error](https://github.com/php/php-src/pull/642) when parsing fractional numbers. The function only retrieves the first 10 decimal numbers and simply chops off the rest, which means that ``` 1701.672119140625 ``` will be parsed as ``` 1701.6721191406 ``` _Notice the loss of the two last fractional digits_ This has luckily been resolved and is [available as of PHP v7.0](https://github.com/php/php-src/pull/993/commits/95cef47afb0b5329915a178d34cf27efcb54607b) which introduced a new option for the function `JSON_PRESERVE_FRACTIONAL_PART`. ``` json_decode($json_data, true, 512, JSON_PRESERVE_FRACTIONAL_PART) ``` Unfortunately this does little for those still stuck on older versions of PHP. Unfortunately Google does not offer [client libraries for PHP](https://developers.google.com/maps/documentation/elevation/client-library). So if those extra decimal points matter then you should seriously consider using a third party library.

Handling chart events in Excel 2010 and newer

Sun, 12 Feb 2017 08:00:00 +0100

Enabling VBA code in Excel to receive and handle chart events is not very straight forward to do. Fortunately there are quite a few useful guides available for this, e.g. [Jon Peltier](http://peltiertech.com/chart-events-microsoft-excel/) and [David J. Brett](http://stackoverflow.com/a/7441360/779521). I recently had to do some custom VBA coding in Excel 2010 and a complete working demo would have been immensely useful. So here is it for anyone that might want to hit the ground running, so to say. ## Get the example workbook First of all you can download the [example workbook here](/files/excel-vba-events-demo.xlsm) (it's macro enabled). > To view the code open the VBA Editor (Alt+F11). ## How it is organized

Organization of the sample workbook code

### The Class Module `CEventChart` This is the main logic where you should put most of your code. This class module acts as the receiver that Excel calls when the charts receive events. Here you need to put the event functions you want to handle (there are examples in there on how to handle both the [Activate](https://msdn.microsoft.com/en-us/library/office/ff834456.aspx) and [Calculate](https://msdn.microsoft.com/en-us/library/office/ff820890.aspx) event). ``` vb Option Explicit ' Declare object of type "Chart" with events ' This object will hold the Chart instance that is currently being manipulated Public WithEvents EvtChart As Chart ' *********** The Excel event functions go below this line, avoid having too much custom code in these functions ' rather have the event functions call them as private functions (e.g. like MyGreatFunction below) Private Sub EvtChart_Activate() ' Executed whenever the chart is enabled (e.g. users selects it) ' TODO: Here comes your logic... MyGreatFunction End Sub Private Sub EvtChart_Calculate() ' Disable screen updating while the subroutine is run. Application.ScreenUpdating = False ' TODO: Here comes your logic... MyGreatFunction 'Enable screen updating again Application.ScreenUpdating = True End Sub ' + any other evens you want to subscribe to, see: https://msdn.microsoft.com/en-us/library/office/ff822192.aspx '************* Your private functions go below this line Private Sub MyGreatFunction() ' You can access the chart object here by using the EvtChart object Debug.Print "Chart: " + EvtChart.Name End Sub ``` ### Module `ChartEventHookups` This module handles the nitty gritty details of connecting/disconnecting the chart events in your workbook. You should not need to modify this code at all and it can be left as is. ``` vb Option Explicit Dim clsEventChart As New CEventChart Dim clsEventCharts() As New CEventChart ' ************************************************* ' This function connects all chart objects in the currently active sheet to the event enabled ' class module CEventChart. This class module contains functions to handle event callbacks raised ' by the chart object when either the user or excel interacts with it. Sub EnableEventsForAllCharts() ' Enable events on sheet if it is a chart sheet If TypeName(ActiveSheet) = "Chart" Then Set clsEventChart.EvtChart = ActiveSheet End If ' Enable events for all charts embedded on a sheet ' Works for embedded charts on a worksheet or chart sheet If ActiveSheet.ChartObjects.Count > 0 Then ReDim clsEventCharts(1 To ActiveSheet.ChartObjects.Count) Dim chtObj As ChartObject Dim chtnum As Integer chtnum = 1 For Each chtObj In ActiveSheet.ChartObjects Set clsEventCharts(chtnum).EvtChart = chtObj.Chart chtnum = chtnum + 1 Next ' chtObj End If End Sub ' ************************************************* ' Unhook the event classes from the charts Sub DisableEventsForAllCharts() ' Disable events for all charts previously enabled together Dim chtnum As Integer On Error Resume Next Set clsEventChart.EvtChart = Nothing For chtnum = 1 To UBound(clsEventCharts) Set clsEventCharts(chtnum).EvtChart = Nothing Next ' chtnum End Sub ``` ### Excel Object `Sheet2 (Charts)` This code should be placed in all worksheets that contain charts you want to receive events for. What this code does is simply connect the events when the sheet is activated first and then disconnect the events when another sheet is activated. This helps save resources and speed up calculations. ``` vb Option Explicit '**************************************************************** ' When the worksheet is selected hook up the event classes. ' If the chart source data has changed in the mean time then the Calculate event will be raised Private Sub Worksheet_Activate() EnableEventsForAllCharts End Sub '**************************************************************** 'When the worksheet is deselected disable the events Private Sub Worksheet_Deactivate() DisableEventsForAllCharts End Sub ``` ### Excel Object `ThisWorkbook` This code handles hooking the chart events up correctly when the workbook is first opened if the workbook is opened with the chart sheet active. ``` vb Option Explicit '**************************************************************** ' When the workbook is first opened then the charts in the active sheet must ' be connected immediately otherwise out of date data might be displayed (in case the book closed unexpectedly before) Private Sub Workbook_Open() ' Disable screen updating while the subroutine is run. Application.ScreenUpdating = False EnableEventsForAllCharts 'Enable screen updating again Application.ScreenUpdating = True End Sub ``` ## Get the example workbook Again, you can download the [example workbook here](/files/excel-vba-events-demo.xlsm) (it's macro enabled). > To view the code open the VBA Editor (Alt+F11).

Combining Mp4 video files automatically

Sat, 14 Jan 2017 08:00:00 +0100

I have been a fan of the [GPAC](https://gpac.wp.mines-telecom.fr/downloads/) video tool package for a while. I've especially found their `mp4box` utility useful when dealing with mp4 files. Adding subtitles, creating chapters, combining multiple audio streams, you name it. This little tool does it all and blazingly fast. See tool However I recently ran into a small annoyance when I had to concatinate multiple mp4 video files into a single big one. I wanted to resulting file to have chapter marks created automatically at the joins between the mp4 files but there was no way to do this using the mp4box tool. So I ended up writing a small python script that automates the process of joining mp4 video files into a single file. This tool can also break the joining up based on the desired output file size (e.g. if you want to burn the combined file to a DVD or BluRay disk and you want it to fit.) The mp4combine is open-source and available on my Github repo _Note:_ You need to download and install the [GPAC](https://gpac.wp.mines-telecom.fr/downloads/) tools first before using this script.

Jekyll 3 Pagination Gem

Sat, 19 Nov 2016 20:00:00 +0100

I recently learned that the true and tested pagination gem for jekyll [jekyll-paginate](https://github.com/jekyll/jekyll-paginate) had been decommissioned and removed from Jekyll 3. This is unfortunate as this gem is very good but lacks only a few major new features, most notibly the feature to paginate on a subset of your posts (e.g. only paginate on certain categories, tags or languages). > Be the pull request you want to see in the world So I decided to dive in and unofficially 'adopt' this part of the Jekyll world and build a version 2 of the pagination gem. And here it is, fully backwards compatible and with a long list of new enhancements https://github.com/sverrirs/jekyll-paginate-v2 # Installing The gem can be simply installed straight from rubygems.org by using ``` gem install jekyll-paginate-v2 ``` More info on configuration on my [Github page](https://github.com/sverrirs/jekyll-paginate-v2#installation) # Using it The [Github project page](https://github.com/sverrirs/jekyll-paginate-v2#installation) goes into all the detailed ins-and-outs of configuring and enabling the features of this gem. My blog now uses this gem for it's pagination logic and I'm really looking forward to finally being able to make the category names into click'able links. This also marked my first Ruby gem ever, check it out on https://rubygems.org/gems/jekyll-paginate-v2 _Be excellent to each other_

Google Analytics and server-side PHP

Tue, 15 Nov 2016 20:00:00 +0100

Google published a handy API to implement hit tracking in your projects, the [Measurement Protocol](https://developers.google.com/analytics/devguides/collection/protocol/v1/). This API is especially useful for APIs and other platforms that cannot use the pre-existing solutions to communicate usage and tracking stats with the Google Analytics servers. # The Protocol The Analytics Mesurement Protocol API is actually pretty simple. You can access it either through a HTTP GET or HTTP POST request and send along a few variables. The host you send your data to is _https://www.google-analytics.com/collect_. > I would recommend using POST as you can run into unforseen caching issues either locally or in proxy servers when using GET requests. The API offers a dizzing amount of [parameters](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters) to choose from. The example in this article mentions only a handfull of the most basic tracking parameters that are relevant when tracking visits and exceptions that your users encounter. # The PHP Code The PHP code is split into two classes. An abstract `AbstractAnalytics` class that serves as a common base for implementing analytics tracking. Then there is the concrete `GoogleAnalytics` class that implements the specifics of talking to the Google Analytics servers. ## The Abstract Base Class ``` perl abstract class AbstractAnalytics { protected $debug = false; public function __construct($debug) { $this->debug = $debug; } /* Sends a normal page-view type tracking request to the analytics server */ public function Track($title) { if (!method_exists($this, "GetHitRequest")) { throw new Exception( "Missing GetHitRequest function"); } $response = $this->_URLPost( $this->getHost(), $this->GetHitRequest($this->getUrlPath(), $title)); if( $this->debug ) echo $response; return $response; } /* Sends a exception type tracking request to the analytics server */ public function Error($title, $errorcode) { if (!method_exists($this, "GetErrorRequest")) { throw new Exception( "Missing GetErrorRequest function"); } $response = $this->_URLPost( $this->getHost(), $this->GetErrorRequest($this->getUrlPath(), $title, $errorcode)); if( $this->debug ) echo $response; return $response; } /* Gets the analytics host name (e.g. https://www.google-analytics.com) */ abstract protected function getHost(); /* Gets the full url to the requested resource */ protected function getUrlPath() { return 'http' . (isset($_SERVER['HTTPS']) ? 's' : '') . '://' . "{$_SERVER['HTTP_HOST']}{$_SERVER['REQUEST_URI']}"; } /* Gets the user agent attached to the original request */ protected function _getUserAgent() { return array_key_exists('HTTP_USER_AGENT', $_SERVER) ? $_SERVER['HTTP_USER_AGENT'] : ""; } /* Gets the http referer for the original request */ protected function _getReferer() { return array_key_exists('HTTP_REFERER', $_SERVER) ? $_SERVER['HTTP_REFERER'] : ""; } /* Gets the remote ip address for the original request */ protected function _getRemoteIP() { return array_key_exists('REMOTE_ADDR', $_SERVER) ? $_SERVER['REMOTE_ADDR'] : ""; } /* Performs a POST request of the data in $data_array to the URL in $url */ private function _URLPost($url, $data_array) { // Need to encode spaces, otherwise services such // as Google will return 400 bad request! $url = str_replace(" ", "%20", $url); // Construct the contexts for the POST requests $opts = array( 'https'=>array( 'method'=>"POST", 'header'=> "Accept: application/json, text/javascript, */*; q=0.01\r\n". "Content-type: application/x-www-form-urlencoded; charset=UTF-8\r\n". "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36\r\n". "Referer: https://api.example.com/\r\n", 'content' => http_build_query($data_array) ) , 'http'=>array( 'method'=>"POST", 'header'=> "Accept: application/json, text/javascript, */*; q=0.01\r\n". "Content-type: application/x-www-form-urlencoded; charset=UTF-8\r\n". "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36\r\n". "Referer: https://api.example.com/\r\n", 'content' => http_build_query($data_array) ) ); $context = stream_context_create($opts); $result = null; $dh = fopen("$url",'rb', false, $context); if( !$dh ) return null; if( $dh !== false ) $result = stream_get_contents($dh); fclose($dh); return $result; } } ``` There are three functions of interest above. The pure abstract function `getHost` which all subclasses must implement, this function returns the HTTP address for the Analytics tracking server. Then it is the public `Track` function that is called to track a regular _hit_ when a user visits one of your service endpoints. This function returns an array of keyed variable that will make up the content data in the POST. Finally it is the public `Error` function that is called to track errors or exceptions that happen in your service endpoints and are presented to your users. Similarly to Track this function returns an array of POST data. ## Concrete Google Analytics Class ``` perl require_once('AbstractAnalytics.php'); class GoogleAnalytics extends AbstractAnalytics { protected $google_host = 'https://www.google-analytics.com/collect'; protected $google_debug_host = 'https://www.google-analytics.com/debug/collect'; /* The Google Analytics Tracking Id for this property (e.g. UA-XXXXXX-XX) */ protected $trackingId = ''; /* The name of the application, this is sent to the Google servers */ protected $appName = ''; public function __construct($TrackingID, $ApplicationName, $debug) { parent::__construct($debug); $this->trackingId = $TrackingID; $this->appName = $ApplicationName; } protected function getHost() { if( $this->debug ) return $this->google_debug_host; return $this->google_host; } private function getCommonDataArray($url, $title){ // Standard params $v = 1; $cid = $this->_ParseOrCreateAnalyticsCookie(); return array( 'v' => $v, 'tid' => $this->trackingId, 'cid' => $cid, 'an' => $this->appName, 'dt' => $title, 'dl' => $url, 'ua' => $this->_getUserAgent(), 'dr' => $this->_getReferer(), 'uip' => $this->_getRemoteIP(), 'av' => '1.0' ); } protected function GetHitRequest($url, $title) { // Create the pageview data $data = $this->getCommonDataArray($url, $title); $data['t'] = 'pageview'; // Send PageView hit as POST return $data; } protected function GetErrorRequest($url, $title, $errorcode){ // Create the error data $data = $this->getCommonDataArray($url, $title); $data['t'] = 'exception'; $data['exd'] = $errorcode; $data['exf'] = '1'; return $data; } // Gets the current Analytics session identifier or // creates a new one if it does not exist private function _ParseOrCreateAnalyticsCookie() { if (isset($_COOKIE['_ga'])) { // An analytics cookie is found list($version, $domainDepth, $cid1, $cid2) = preg_split('[\.]', $_COOKIE["_ga"], 4); $contents = array( 'version' => $version, 'domainDepth' => $domainDepth, 'cid' => $cid1 . '.' . $cid2 ); $cid = $contents['cid']; } else { // no analytics cookie is found. Create a new one $cid1 = mt_rand(0, 2147483647); $cid2 = mt_rand(0, 2147483647); $cid = $cid1 . '.' . $cid2; setcookie('_ga', 'GA1.2.' . $cid, time() + 60 * 60 * 24 * 365 * 2, '/'); } return $cid; } } ``` This class collects and returns the Google Analytics specific variables for both the Track and Error scenarios. This class automatically switches between the Validation Server if the `$debug` parameter is set to `true`. This is very handy when testing your measurement configuration as the validation server will return you JSON data with information about your request. Normally the Analytics servers do not give you any response data back. And that is it. You can now instantiate your class and start tracking like so ``` perl $analytics = new GoogleAnalytics("UA-XXYYMM-X", "My API", true); ... $analytics->Track("My Awesome API function"); ... $analytics->Error("NonExistingEndpoint", "404"); ``` Hope you find this useful, please leave comments below if you have any questions or issues.

Writing a Python webserver in 10 seconds

Fri, 04 Nov 2016 20:00:00 +0100

Today I [learned](http://stackoverflow.com/a/5128451/779521) that you can run a simple local webserver for static content with only a single line ``` python python -m http.server [] ``` That is simply awesome! It will serve all available files and even list directory contents for you with no problems. However this basic [http.server](https://docs.python.org/3/library/http.server.html) is very simple and you will quickly run into one of it's biggest drawback which is that it can only handle a single request at a time.

Super adorable, but not very efficient.

Choking on more than one file at a time is not very desirable if you want to be able to make multiple requests within a short time period or if you want more than one user to access the server at a time. # The 10 minute version Extending the http server code to work for more than a single request at a time is still not very difficult. Below is a fully functioning example of one ``` python #!/bin/env python import sys, signal import http.server import socketserver # Reading portnumber from command line if sys.argv[1:]: port = int(sys.argv[1]) else: port = 8080 # Note ForkingTCPServer does not work on Windows as the os.fork() # function is not available on that OS. Instead we must use the # subprocess server to handle multiple requests server = socketserver.ThreadingTCPServer(('',port), http.server.SimpleHTTPRequestHandler ) #Ensures that Ctrl-C cleanly kills all spawned threads server.daemon_threads = True #Quicker rebinding server.allow_reuse_address = True # A custom signal handle to allow us to Ctrl-C out of the process def signal_handler(signal, frame): print( 'Exiting http server (Ctrl+C pressed)') try: if( server ): server.server_close() finally: sys.exit(0) # Install the keyboard interrupt handler signal.signal(signal.SIGINT, signal_handler) # Now loop forever try: while True: sys.stdout.flush() server.serve_forever() except KeyboardInterrupt: pass server.server_close() ``` _The code above is based on an example created by [Kevin Decherf](https://kdecherf.com/blog/2012/07/29/multithreaded-python-simple-http-server/)._ This server can be run directly from the commandline similar to the http.server and with an optional port number as well (assuming the code is stored in the file httpserver.py) ``` python httpserver.py 8080 ``` This server is actually a pretty good light-weight server for static content. For simple web content such as HTML, JavaScript and stylesheets it works really well. As an in-place testing server I would recommend it. However it doesn't handle serving large files such as videos and audio very well.

Too big for the inter-connected-network-of-tubes.

# Serving large files over HTTP In the case where you want a simple webserver that handles serving both simple static content and streaming video and audio files I must recommend using the [node.js](https://nodejs.org) [http-server](https://github.com/indexzero/http-server) module. It is simply superb. Assuming you have both node.js and npm installed then it is simply a matter of ``` npm install http-server -g ``` And then you can run your webserver directly from the command window like shown below. Just specify which directory you want it to serve content from and the port and you're done ``` http-server "C:\www\mywebsite" -p 8080 ``` This server handles serving large files (>300MB) quite well, mostly due to the async way that node.js works. But I'm not complaining. ...and it goes really well with that [home-made media server](/2016/10/html5-video-playback-xbox.html) website you built!

You made a remote as well right?