Nailing Jelly

York Bin Collection API Update

Thu, 30 Dec 2021 16:49:00 +0000

Sometime in 2021 the API I was using to collect information on Bin Collection in York stopped working, and it took me an inordinate amount of time to get to finding out what had happened.

It appears the API I was using was replaced by a simpler one - although the documentation appears to say that this one has been the one since 2016…

So I have (finally), updated the shim I used to pull the data.

The updated shim can be found in the Github repository at https://github.com/nigelm/york_bin_collection

Freeserve - How We Built The System

Tue, 06 Jul 2021 00:00:00 +0100

“Success has many fathers, but failure is an orphan”

There have been many people who have claimed to be “The Inventor of Freeserve”, although in many cases it appears the more fuss they make about this title the less they had to do with it… I wasn’t in the room when that particular proposal was made (although I have quite strong suspicions on who originally proposed the idea), but I was on the original implementation team within Planet Online.

I mostly worked in the Core Systems group at Planet between 1995 and 1999. As part of that I worked on the CAG (Connect and Go) systems which provided a white label dial up ISP service for the likes of Prestel, Kingston (the Hull telecoms provider) and Channel 6 which was a predecessor to Freeserve.

And then I worked on Freeserve. In particular the sign up and authentication system were mine, as well as the mail system. However this particular reminiscence is going to concentrate on the mail systems.

What Was Freeserve

Freeserve was a consumer ISP originally run by Dixons (the high street tech chain) with Planet Online. Dixons gave a huge advertising and high-street footprint - they had a store on every UK high street where you could pick up your Freeserve floppy disk or CD.

Although Freeserve was not the very first to offer no-fee dial-up internet service, the clout of Dixons and the advertising campaign meant that it took off at a rate of knots, putting on 900,000 accounts in the first 4 months after its launch in 1998.

The service initially ran over dial-up modems, using a national dial up number which was free of charge to the initiator. Through the magic of the complicated UK telephony cross-charging system this provided a small termination payment to the service provider - and this was used to pay for the service. There were other revenue streams - advertising, premium technical support and later pay for services.

Why Now?

I was talking to a couple of friends this morning, and Freeserve was mentioned. My friend was incredibly enthusiastic about the Freeserve mail system - how you could set up different email addresses within the domain and have a set of filtering to individual family mailboxes - in 1998! And I had to say “I wrote that”.

Last year was 25 years since we started Planet Online - I was one of the first 3 hires there. But, and especially with the Covid situation, that anniversary passed unnoticed. Its 22 years since Freeserve launched and made a significant change to the UK home internet market - in the last couple of years the last vestiges of the Freeserve name and service have been finally closed off.

Freeserve Email

Freeserve email addresses were of the form user@example.freeserve.co.uk - the user part could be anything, and it was just the example.freeserve.co.uk part that identify the Freeserve account. All messages to any user @example.freeserve.co.uk would end up in one single POP3 mailbox.

As was normal for the time, email was provided to the customer over POP-3, with outgoing email sent by SMTP via a Freeserve mail server. To reduce SPAM and other shenanigans there was a route map applied to the Freeserve terminating modem racks which forced all SMTP connections to a local (to the modem rack) SMTP server.

The mail receiving stack was Exim delivering into a Maildir mail spool which resided on a set of NetApp NFS filesystems. This gave us a huge (for the era), mail spool with fairly high availability. There were some performance tweaks applied to make this work for the size of mail spool and loading.

The Maildir+ specification allows additional tag information to be added to a message by tweaking the filename that the message is saved under. The configuration used saved both the message size and the local recipient address into these tags.

The mail spool was NFS exported to a number of the mail servers - which were all Linux systems. The pop side was the qmail qmail-pop3d with some local modifications and an additionally modified qmail-popup using a locally built authentication utility to replace checkpassword.

The checkpassword replacement queried the Freeserve RADIUS servers for authentication. Those had some additional performance tweaks which I may come to another day.

The qmail-popup program collected the pop3 username and password. The username would be the email domain - ie example.freeserve.co.uk - however you could also specify an email address - ie user@example.freeserve.co.uk - and some other patterns. These resulted in additional information being put into the environment so that this data was available to the qmail-pop3d later on. Authentication was passed onto the checkpassword binary.

The main part of the POP daemon was qmail-pop3d - this had several modifications:-

The size of each email is taken from the tag part of a Maildir message filename - this saved an additional stat() operation to check the mail size, which meant that for a NetApp filer all the information needed to produce a POP LIST was in the NetApp fast data.
There was support for a broadcast email setup where previously unseen messages in a global mailbox are linked into the users own email. Not sure if this was ever used but it allowed a space efficient way to send an email to all.
The additional environment information from qmail-popup was used to filter the view of the mailbox - so if this had been set to user then only emails sent to user@... would be shown in the POP3 listings. This also used the additional information put into the Maildir+ tags to avoid having to read or stat the individual message files.

Performance Tweaks

As described above, the pop daemon was tweaked to only stat or touch mail files when absolutely necessary - basically when a message was actually read by the client (after which the status would be updated by renaming - thats Maildir semantics).

The Mail directory was hashed to spread the layout across the filer - this could have been extended across multiple filer heads, but that was not necessary during the time I was there.

The mail hosts always attempted once to deliver external mail. After that they passed the mail to a fallback routing host which dealt with all the things that could not be delivered immediately. This kept control of the size of the mail queues on eveerything except the fallback host (which had different queue running parameters on it).

Security Tweaks

Exim was running entirely as a single user, no setuid etc, this reduced the possibilities for security exploits (although this also meant that Exim could see the entire mail spool - but that was a relatively low risk problem at that time).

The qmail side chrooted to the mail spool for a user on authentication.

Due to expected (and seen) issues with spamming (from our users), and attempting to remain a good internet citizen, Freeserve blocked SMTP from end users out of the network. Even more so (and controversially), we routed all SMTP connections to a local (to the modem rack pop) SMTP server. This routed all outgoing mail. It also had a view on all mail passing through it, and used some very simple heuristics to pick up spam attempts - any SMTP connection that attempted to push more than a few messages out would have the mail queued for later checking. This checking had more context on what was happening outside the single mail connection, so could pick up mail bombing runs (there was a craze for “Yo Momma” emails in the first months of Freeserve).

The small number of people who hit the abuse filters had their mail blocked, and after a small number of repeat offences had their username block and their calling line ID (which we always received because it was required for the telephony call class) was blocked.

After initial hysteria from the previously established large ISPs they did grudgingly recognize that Freeserve was not a bad spam source.

Notes

Originally I wrote this back in January, but hadn’t quite finished it… and then the post lingered for 6 months. So I have now hurridly finished it off and pushed it out because improving it further might mean it could wait another 6 months…

Python Development Environments

Sat, 24 Oct 2020 17:47:31 +0100

I have spent a substantial number of years mostly programming in Perl, but this year I have been working on a team that has a strong preference for Python and so I have converted across.

Previously I have dabbled in Python - I could read basic python, and had patched various things - mainly Ansible - in the past, but had not done any serious work in the language.

However its not the transfer to the language thats caused most of the headaches, but other things surrounding that - for example:-

Development environment
Formatting, checking, rules for how things go into git
Documentation

We don’t have much in the way of stated rules for these, and despite Python being much more about there being one right way to do things, in terms of overall development environments it all appears that there is a huge variety of ways that people do stuff…

So this is what I am now moving to for things I am working on…

Development Environment

I use Poetry. This brings together management of virtual environments, dependency lists etc as well as distribution builds and pushing of packages to pypi or to a local repository.

The downside is that you tend to have to run things by poetry run cmd..., but I mostly set that up in a Makefile to deal with that. Additionally at present the cost of running builds and deploys in CI is bloated by the additional time to install poetry - I may build myself an appropriate Docker image to ameliorate that.

The Makefile I add is hand crafted - mostly cargo-culted from that last thing I did - but I now use the self documenting Makefile trick to add a set of help to the Makefile.

Version number handling is managed by bump2version - using that to rewrite the poetry version tag as well as updating the information within the package __init__.py file (yes this may well be superfluous but the alternatives appear to require loading additional modules just to find your own version number). Additionally I found a hack this last week to make bump2version update the CHANGELOG.md file by using the alternate methods of Markdown formatting (otherwise the # header markers cause the bump2version config file to ignore the lines as a comment).

So in ‘.bumpversion.cfg’ you have:-

[bumpversion:file:CHANGELOG.md]
search = <!-- insertion marker -->
replace = <!-- insertion marker -->
        [{new_version}] - {now:%Y-%m-%d}
        --------------------

and in CHANGELOG.md you have:-

Unreleased Changes
------------------

<!-- insertion marker -->
- last commit changelog entry

Pre-Commit Formatting / Checking

I have a .pre-commit-config.yaml file in each repository - mostly exactly the same, but with some per-project tweaks. I run pre-commit prior to each commit - mostly not currently enforced by actually adding it to git hooks, although I may change that as I did previously have a set of hooks set up by git-hooks to run the perl TidyAll

A sample .pre-commit-config.yaml file can be found here, but in general I am running a set of checks including the black formatter, flake8 and mypy.

In terms of git management I currently tend to work in Fork having previously used Tower for many years (Tower is likely a bit more powerful and slick, but it now costs more per year than Fork does outright - although I hope that Fork will not regret going for a one off cost model).

Documentation

Documentation is another thing that seems odd regarding python - there is the start of a standard workflow defined with docstrings etc, and then things go a bit freestyle. I have to say I am not very keen on reStructuredText, and Sphinx felt opaque to me - and then you hit several different API documentation mechanisms, many of which appear to be machine rather than human readable. This is one of the areas where the Perl simple but consistant documentation setup works better - although the python side does pull more introspection information out of the code.

So I have settled on one I like (so far - only been working in this for a few weeks) and that appears to work well with my other tools.

I use MkDocs. With that I have been using the Material for MkDocs theme. Stacked on top of that are the mkdocs-click extension for CLI tools, and mkdocstrings for API documentation.

There is one little problem with the pipeline above in that mkdocs-click and mkdocstrings use different function signature markup - this is something I mean to have a look at in the future.

CI etc

The CI workflow varies not least because I release some things on Github using Travis CI, and other things are handled by an internal GitLab instance using GitLab CI. However in both cases I have got a fairly standard test, release and build documentation workflow with the artifacts going to the appropriate repo and documentation going to Github/GitLab Pages.

I am not currently doing a lot of additional multi-version testing with tox etc. I may move to this later, but am trying to take this a step at a time.

Home Assistant - Bin Collection Display and Notification

Sun, 03 May 2020 11:28:55 +0100

Previously I had built a shim to make it much easier to import data from the York Bin Collections API into Home Assistant

I am making use of this data in two ways:-

Making a display card in Lovelace to display information about the next bin collection.
Making a notification to prod me to put the right bins out.

Lovelace Display

This is currently a prototype - and I need to spend more time on working out exactly what I want from the Home Assistant display - I may move to building a substantially minimised display for use with a spare Nook that I have - in a similar way to Turn an old eReader into an Information Screen (Nook STR)

The display card looks like this

And this is the result of adding a Markdown card, setting the title to Bin Collection and filling the card in with some content. The overall YAML definition of the card (click the Show Code Editor link on the card editor) is:

content: >2-
   {% set bin = 'sensor.bin_collection' %}
   {% set nextdate = strptime(states(bin),'%Y-%m-%d') %}

  Next collection on {{ nextdate.strftime("%A (%-d %B, %Y)  ") }} {%- if
  'blackbin' in state_attr(bin, 'next_collection_types') %}<ha-icon
  icon="mdi:delete"></ha-icon>{% endif %} {%- if 'box' in state_attr(bin,
  'next_collection_types') %}<ha-icon icon="mdi:recycle"></ha-icon>{% endif %}
  {%- if 'greenbin' in state_attr(bin, 'next_collection_types') %}<ha-icon
  icon="mdi:pine-tree"></ha-icon>{% endif %}


  Items collected:- {% for set in state_attr(bin, 'next_collection_types') %} {%
  set thing = state_attr(bin, set) %}

  - {{ thing.MaterialsCollected }} {% endfor %}
title: Bin Collection
type: markdown

It is incredibly sensitive to initial spacing and indents…

The first two lines of the content:-

{% set bin = 'sensor.bin_collection' %}
{% set nextdate = strptime(states(bin),'%Y-%m-%d') %}

set the initial config up.

The next block outputs the next collection date in a reasonable format, followed by icons for the appropriate bin type(s) - this is rather complex in setting up the right icon to display.

The final chunk lists the types of material being collected this week - it iterates through the types of collection for this week, and pulls the information this collection into the thing variable. Thats then used to fill in the list elements.

This chunk may well be made more detailed in the future, but for now its a prototype, and theres a lot of other things than can be built on this should someone desire a different presentation.

Bins Out Notification

The intention here was to send a nudge, on the evening before the bins are collected, to remind me to put the appropriate bin out.

Notifications in Home Assistant are most easily handled by an Automation. I tend to use Pushover notifications, but there are various other Home Assistant Notifcations.

The complete automation used looks like this:-

- id: '1588360498442'
  alias: Bin Day Tomorrow Notification
  description: Tell me about bin day tomorrow
  trigger:
  - at: '19:15'
    platform: time
  condition:
  - condition: template
    value_template: '{{ as_timestamp(now())+86400 > as_timestamp(states(''sensor.bin_collection''))
      }}'
  action:
  - data_template:
      message: '{% set bin = ''sensor.bin_collection'' %}

        Items collected:- {% for set in state_attr(bin, ''next_collection_types'')
        %} {% set thing = state_attr(bin, set) %}

        - {{ thing.MaterialsCollected }} {% endfor %}

        '
      title: Bin Day Tomorrow - {{ ', '.join(state_attr('sensor.bin_collection', 'next_collection_types'))
        }}.
    service: notify.pushover

So this is an automation that runs every day at 19:15 (the trigger: section), with a condition to check if the collection is tomorrow (see if now as a Unix timestamp with 1 days worth of seconds added, is greater than the timestamp version of the bin collection date - this is in the condition: section).

The action: section calls up the Pushover notification with a message and title mostly based on the Lovelace card information shown above.

Minor Gardening

Sat, 02 May 2020 19:50:53 +0100

I’ve done some tidying of the old posts - mostly this should have zero effect on what you see, but the Markdown has been tidied (line wraps added in to make editing more comfortable), and any internal links should now be internal rather than going back to the previous wordpress site.

Now to actually add some current content…

York Bins API and Home Assistant

Mon, 20 Apr 2020 17:05:43 +0100

I have been using Home Assistant to automate various things at home. One of the things I thought would be useful was the ability for the system to inform me of when the next bin collection is, and what sort of collection this would be.

Thankfully York Council has an API telling you about their bin collections. This is documented at York Waste Collection Lookup

Home Assistant REST Sensors

Home Assistant helpfully can use REST sources as a sensor. Unfortunately the formatting of the data in the York API is not well arranged to be easy for Home Assistant to use - in particular Home Assistant is not easily able to handle returned data that is organised as an array of dictionaries.

People have made use of this in the past - for example see this example. However this makes multiple calls to the API, which is unfriendly, and relies on the API always returning its data in the same order in the array, which appears to not be specified.

Python Command Line Shim For Home Assistant

I decided it was better to put a shim in to pull the data from the API, do a little processing on the data - specifically changing the date format into something sane - and arrange these as a dictionary with some additional overall collection data around it.

This shim can be found in the Github repository at https://github.com/nigelm/york_bin_collection

A sample of using it would look like:-

sensor:
  # York Bins Collection API - 3 sets, 1 for each bin
  - platform: command_line
    command: /config/york_bin_collection.py 100050567115
    name: Bin Collection
    scan_interval: 86400
    value_template: '{{ value_json.next_collection }}'
    json_attributes:
      - next_collection
      - next_collection_types
      - blackbin
      - greenbin
      - box
      - updated

and this would return the date of the next collection as the main value - currently 2020-04-24 - and the state attributes (reported as YAML) then looks like:-

next_collection: '2020-04-24'
next_collection_types:
  - blackbin
blackbin:
  BinType: GREY 180
  BinTypeDescription: Grey Bin 180L
  CollectionDay: FRI
  CollectionDayFull: Friday
  CollectionDayOfWeek: 5
  CollectionFrequency: Alternate Weeks
  CollectionFrequencyShort: WEEK 2
  CollectionPoint: FRONT
  CollectionPointDescription: Edge of Property at Front
  CollectionPointLocation: null
  CollectionType: GREY BIN/SACK
  CollectionTypeDescription: Grey Bin/Black Sack Collection
  ImageName: blackbin
  Locality: null
  MaterialsCollected: General Domestic
  NumberOfBins: '1'
  WasteType: GREY BIN/SACK
  WasteTypeDescription: Grey Bin/Black Sack Collection
  last: '2020-04-10'
  next: '2020-04-24'
greenbin:
  BinType: GREEN 180
  BinTypeDescription: Green Bin 180L
  CollectionDay: FRI
  CollectionDayFull: Friday
  CollectionDayOfWeek: 5
  CollectionFrequency: Alternate Weeks
  CollectionFrequencyShort: WEEK 1
  CollectionPoint: FRONT
  CollectionPointDescription: Edge of Property at Front
  CollectionPointLocation: null
  CollectionType: GREEN
  CollectionTypeDescription: Green Collection
  ImageName: greenbin
  Locality: null
  MaterialsCollected: Garden Waste
  NumberOfBins: '1'
  WasteType: GREEN
  WasteTypeDescription: Green Collection
  last: '2019-11-29'
  next: null
box:
  BinType: BOX 55
  BinTypeDescription: Box 55L
  CollectionDay: FRI
  CollectionDayFull: Friday
  CollectionDayOfWeek: 5
  CollectionFrequency: Alternate Weeks
  CollectionFrequencyShort: WEEK 1
  CollectionPoint: FRONT
  CollectionPointDescription: Edge of Property at Front
  CollectionPointLocation: null
  CollectionType: KERBSIDE
  CollectionTypeDescription: Kerbside Collection
  ImageName: box
  Locality: null
  MaterialsCollected: 'Paper/Card : Plastic/Cans : Glass'
  NumberOfBins: '3'
  WasteType: KERBSIDE
  WasteTypeDescription: Kerbside Collection
  last: '2020-04-17'
  next: '2020-05-01'
updated: '2020-04-20 16:25:55.918518'
friendly_name: Bin Collection

This data can then be used in other automations - thats what I am going to look at next.

Relocation and Resurrection

Sat, 18 Apr 2020 00:00:00 +0000

After a vastly long hiatus (the post 5 years back stating I intended to blog more really doesn’t count), I’ve finally put something together.

Basically I have a batch of things I am trying out and so want to keep more permanent notes on, hence bringing this back into play.

Hosting

Previously I put content onto wordpress.com – I certainly did not want to get involved in actually managing the blogging platform. However in the mean time I have been using Hugo for various thing, and it is a much nicer system for me to work with, so moving to Hugo was an easy choice.

Additionally I can now use Github Pages to host the content (or various similar services).

The old Wordpress content was dumped as an XML export and converted to Markdown as described in Conversion tools from Wordpress to Hugo

The theme is hugo-flex

Now all I need to do is write more content.