Categories
Startup World Wide Web

Eukles: Marketing Automation done DIY

For the past couple of years I have been playing around with the idea of a hard divide between outgoing communication (transactional emails, push notifications, newsletter contact list synchronization, marketing automation, slack notifications, etc) and the business logic of my applications.

It never made sense to me that a backend system that manages ticket sales for an should would also implement a mailer queue; but also it doesn’t make sense that the copywriter who wants to change the content of an email would have to talk to a developer to do so.

In 2017 I started working on removing all logic for these kind of actions from my websites and replace them with a single event driven microservice that would handle anything related to outgoing communication for all projects I build.

My goals where to:

  • Be vendor independent: Services like Mailchimp, Moosend, Sendpulse etc have basically already built what I wanted to have, but given my DIY nature I wanted to stay independent from them.
  • Base all actions from a stream of events sent by the business logic.
  • Manage all GDPR related permissions and preferences from a central location.
  • Cut all links between the app domain and this microservice: no accessing data that has not been provided. Push only.
  • Be able to synchronize lists of users (= segments) according to relationships and synchronize these to newsletter services. (ie: automatically make an email list of all registered users who are attending a certain quiz).
  • Provide a management panel where non coders could change the content and the logic of the outgoing notifications.

That is what I started building. I even setup a website to ask feedback from smarter people to make sure I wasn’t reinventing a square wheel, but in the end I kept building nevertheless: Eukles was born.

The Slack notification

I started with our ticking website CatLab Events, and my first goal was pretty modest: send a slack notification to our channel every time someone bought a ticket for one of our quizzes. I setup a MySQL database (probably not the best fit, but as I wanted quick results I stayed with the tools I knew best) and a Laravel/Charon REST API and also created a separate API client package for PHP to easily implement in existing projects. A few days later I registered my first event:

/**
 * @param OrderConfirmed $e
 * @throws \GuzzleHttp\Exception\GuzzleException
 */
public function onOrderConfirmed(OrderConfirmed $e)
{
    $order = $e->order;

    $attributes = [
        'event' => $order->event,
        'order' => $order,
        'ticketCategory' => $order->ticketCategory
    ];

    if ($order->group) {
        $attributes['group'] = $order->group;
        $attributes['subject'] = $order->group;
    } else {
        $attributes['subject'] = $order->user;
    }

    // Track on ze eukles.
    $euklesEvent = \Eukles::createEvent(
        'event.order.confirmed',
        $attributes
    );

    $euklesEvent->unlink($order->user, 'registering', $order->event);

    if ($order->group) {
        $euklesEvent->link($order->group, 'attends', $order->event);
        foreach ($order->group->members as $member) {
            if ($member->user) {
                $euklesEvent->setObject('member', $member->user);
            }
        }
    }

    \Eukles::trackEvent($euklesEvent);
}

To break it down: what I wanted to communicate to Eukles was:

  • An event.order.confirmed event has happened
  • This event has some related properties (event, order, ticketCategory, group and subject (which can be user or group, depending on the event type).
  • Unlink the ‘registering’ relationship between the active user and the event (= the user is not registering anymore, as the registration is already done). The idea is that these relationship actions could be defined either in the business logic or triggers on Eukles itself.
  • Link the ‘attends’ relationships between the group and the event. This would allow us to create a segments of all groups that attend a specific event and synchronize that to Mailchimp later.
  • Add all members of the group as properties of the event (= when a group has 4 members, send an email to every single one of them)

(Note that all models used as properties would have to implement an ‘EuklesModel’ interface that included methods to serialize the content that is send to Eukles.)

The content of the event properties is stored in two places:

  • As part of the event, so that we always know the state of a model at the time of the event creation.
  • (If a unique identifier is provided) the model it is also synchronized with a centralized project-model database that contains the relationships between project-models and always contains the last known properties. Every time an event is sent, the content of these models get updated.

Now I just had to add a SlackService, store authentication details for these services in the database and setup a few Action and Trigger models that would listen for the event and cary out actions. And using the Jobs queue in Laravel everything was handled without impacting the user flow.

Internal use only (for now). Everything is manageable from a REST API with the goal to build an angular frontend in a later stage.
Humble beginnings.

To keep things simple I used Laravels Mustache syntax to parse the arguments that were used by the service clients.

GDPR consents

As Eukles would become responsible for all outgoing communication, I found it fitting to store communication preferences and consents on Eukles instead of in the business logic. Eukles doesn’t really have the concept of a ‘user’, as user project models are sent in events and follow the same logic as any other event property. So I created an API endpoint that would show me all consents in a given project by a given project model. By adding this call to my registration process I was able to ask all users that logged in (existing or new) to give the required and optional consents.

By adding these as filters to segments and actions, I was able to block communications to users who opted out of these messages.

Segments

The goal of Segments was to make dynamic or static lists of project models and synchronize those lists with newsletter services like Mailchimp. For example: by defining segment rules and setting a parent model in the segment schema, Eukles is able to create a segment for each ‘event’ projectmodel, containing all ‘user’ projectmodels that have the ‘attending’ relationship set to it AND have given consent to receive newsletters.

Once every hour these segments are then synchronized with Moosend or Mailchimp, and whoever is in charge of sending out the newsletters can rely on those lists being up to date at all times.

This synchronization also checks for users who are marked as ‘unsubscribed’ in Mailchimp, and the applicable consent is removed from those specific users, resulting in remove them from that segment.

Templates

As all emails in QuizWitz use the same basic template, I have added the ability to define mustache-like ‘templates’ for each project. Anywhere parameters are parsed, these templates can be used (but they only really make sense in html-like contexts)

The email that is sent whenever a player starts a QuizWitz PRO game uses the standard ’email’ template.

Wait 1 hour. if !subject.hasLicense(event.license): then: send email.

And that is pretty much where the project stayed for a few years. My simple event -> trigger -> action model worked fine for sending transactions emails, notify us about certain events and even sending tweets with giphies based on event attributes (implementing services is trivial since the base architecture for defining actions and storing credentials is already provided).

All was well in Eukles world… until I found myself with a bit of time on my hands and I decided to implement some dark design patterns to try to raise the conversion rates in QuizWitz: when someone initalizes a license purchase but in the end decides not to complete the purchase, send them an email asking for feedback.

A perfect use-case for Eukles, where it not that I had gone for the quick event -> trigger -> actions design instead of a fully fledged ‘workflow’ state machine. So that is what I started building.

Instead of triggering an action, all events now check for ‘workflow triggers’ that initializes states in the workflow machine. Depending on the trigger configuration, Eukles also checks for duplicates to make sure that recurring events don’t cause too much triggers. A state then follows the configurable steps in the workflow and triggers actions that are defined in steps. Each trigger also has the ability to set one of the event properties as ‘subject’ that can be used anywhere in the workflow, to reduce the dependency on the initial event.

A simple shunting yard algorithm offers me the ability to define conditions in string format and makes it possible to setup decision trees in the workflow, while a special ‘wait’ step adds a delay to the state machine processing. The whole thing is using Laravel Job queues, so it was all fairly trivial to setup.

The table based interface is obviously a horrible choice for setting up workflows like this, but as for the moment I am the only one using this, it will do. No need to pour more hours in things that only marginally improve my life 😉

And that basically is the current state of the Eukles project. My goal is to clean it up and release it under a GPL license in the future, but in order to do that I first want to improve the admin panel interface and write down some documentation. If that ever happens, I’ll definitely post it here.

Categories
Uncategorized

QuizWitz / Quizfabriek ecosystem

Hello, I am Thijs and I have a problem: I keep buildings things. Silly things. Things that I am sure nobody else will ever use. And yet I keep building them (and release the source code). Why? Because I love it so much. Because I keep thinking of cool use-cases for linking things together.

My story starts in the summer of 2013. I wanted to found a game studio and I had a brand new, original idea for an amazing game: Kahoot. Or… well… our version was called ‘Quizted’ and was aimed towards gamers instead of educators, but the main premise was the same: an audience participation game around quizzing. (Let me spare you the googling: Kahoot beta was released later that year).

I managed to secure some money from the Flanders Audiovisual Fund (VAF) and in the summer of 2014 I hired 3 of the worlds best programming… students. Two software developers (Katia & Yannick) and one marketing student (Ken). Second years students, looking for a summer job. I also hired a consultant from Portugal who would be working on the quiz editor.

By the end of summer, we had a pretty basic proof of concept: we had systems in place to create quizzes, play these quizzes with up to around a hundred people, and I had even developed a rudimentary licensing system to accept online payments. Ken had been working on a crowdfunding campaign and we organised a few ‘quizwitz viewing parties’ where we would play the game with a group of people. I hacked together a QuizWitz mod that allowed a ‘quizmaster’ to control the pace of the game and added a jury mod that allowed manual correction of open ended questions… and our QuizWitz Live events were born. And thus we started our transformation from a gaming studio to an event organiser.

I split up the organisation and moved the event management side into a separate entity (Bektoe vzw, more commonly known as De Quizfabriek). More people joined and we started disrupting the Flemish quiz space with our digital quiz experienceâ„¢. I would be in charge of the quiz software (through my own company), the content would be provided by our fellow members.

It’s all fun and games … until someone has a new idea. And there were ideas. Tons of ideas. Initially we sold ticket through an online form and quizteams had to pay their entrance fee at the door. Due to a whole bunch of no-shows, I replaced that with an online registration system: CatLab Events was born (source code on github). And since we wanted to support the UitPAS program, obviously that had to be integrated as well.

In 2019 our events started to grow unwieldingly large and we started having trouble with handling the bar. It’s not a great expience if you need to wave for ten minutes until a server notices you and then has to disturb the people around you by taking your order. So I wrote a very simple QuizWitz mod that would allow players to order drinks from their table. CatLab Drinks was born (source code available on github).

Everything ran smoothly from their on… until I wanted to experiment with digital drink vouchers. Up until then we had been working with the traditional paper drink vouchers had to be purchased at a central cash registry, but that all felt a bit backdated. So I extended CatLab Drinks to be able to connect to a Raspberry Pi (over websocket) to a service that provided NFC reader/writer capabilities. One of the bigger challenges was that I wanted to store the user credit on the cards themselves, so I would be able to read (and change) the balance even if the central system would go down.

We started a few phased release tests but ultimately started using the system for all our events. I also extended the app even more to allow players to popup their cards using a QR code on the back (resulting in a whole bunch of synchronization challenges that had to be handled every time a card was read or written) and instant payment of remote orders so that servers could just rely on the system instead of manually scanning the cards. Challenge completed. Next!

At one of our themed quizzes we wanted to add a photobooth where players could take a team picture (source code available on github). Taking the pictures was easy, but keeping track of who was on the picture provided to be a challenge. But a solution was at hand! Since every team already had an NFC card, it would be trivial to make a small Raspberry PI based system where players would scan their NFC card and a picture is taken automagically.

Manual picture (people in picture unknown)
Automatic picture: ‘Quizfabriek team’. I must admit that I’ve never figured out how to make the camera focus properly, but that was out of my tech savy scope 😉

But a new problem popped up: the NFC cards were basically anonymous and some teams even used the same card for multiple events. So I had to extend CatLab Drinks even more, to be able to check players in and connect their card to their name and email address. Then I just had to provide an API for the photobooth application, connect the whole bunch up to Eukles (my marketing automation system that also handles all communication from QuizWitz and the ticketing system) and bam, photos were automatically emailed to the players.

Oh, and also all assets, photos, video fragments, etc are stored in a central storage system that makes sure that no duplicate data is stored (source code available on github).

Oh, and I also have a laptop that listen to a datastream (websocket) and emits MiDI signals on certain game events to a digital lighting console to control the theater lights over DMX. But that is not included in the overview.

This is when I learned that an introduction can be too long.

And that is basically what our system looks like. From registering for our events, over checking in and playing, to checking your score on our website when the game is done… An ecosystem of overengineererd madness… But so much fun to build. I’m trying to cut back on new ideas and perhaps properly document what has already been created. Maybe someone on GitHub can see some use of projects after all…

But what if, instead of a quiz, we would gather with 160 players to play a digital escape game…