Bots at Work: Code less, design more.

Overview

It's quite cool to have a virtual personal assistant on your phone to open apps, but things get even better when they can literally do your job. While working as a freelance web developer, I often wondered about the possibility of working remotely and just instructing my device about a design instead of worrying about the coding process. I spent some time contemplating how I can make this a reality and got working.

Brainstorming

My vision was that I would be looking at a live preview of a website, speaking out a modification or instruction and seeing it magically update in front of my eyes. It would be nice if I could hear a response back from the bot (like Siri) if the task was completed or if more information was needed from me.

The conversation could look like:

Me: Add an image to the first section

Bot: Sure! What image should I use?

Me: *Pastes an image link from the web*

Bot: Got it, added the image. Anything else?

Development

The Implementation

1. Styling

First, I knew it would be hard to make the algorithm write both HTML and CSS code, so I decide to make a stylesheet library inspired by Bootstrap. Enter Manuscript.css.

This pre-written stylesheet that saved me time and effort styling each component. It helps anyone add simple and professional UI components to their websites. I spent about a week to make it as customizable as possible and made sure there were several design options and themes to work with.

This library allowed me to style components through HTML itself by compounding classes that contained pre-written styles.

2. Generating Code

I wouldn't go through the entire process of coding a bot that understands natural language to write HTML code because I think it'll be long and boring. Here's a quick summary for the same:

I wrote functions that allowed manipulation of HTML code by considering the code as plain text. These functions required certain parameters to be known (arguments) such as the HTML tag to target, the action to perform, the classes involved, the text to be added, the positioning of the element and more.

3. Understanding intent from natural language and context

The next task was to extract the required information from a natural language instruction and supply that to the appropriate function that could perform the intended action. I implemented this using a standard word to vector conversion and hard coded a heuristic approach that used the presence of keywords and their position in the sentence to run a specific function. (I guess it's not the best way to do this, but certainly effective given that I was manually creating a dataset and vocabulary for the bot.)

Additionally, one could not expect every instruction to contain all the information, so some form of memory must be added to the program to remember context through past instructions. This was done by maintaining a stack of operations performed and using previous instructions to determine new actions. This information is also used to reply with relevant questions to obtain missing pieces of information.

Demo

The Interaction

If you wish to know more about the process, you can write to me or talk to me about it and I would be happy to share more details. I think the more interesting part is how the interaction with the bot takes place. To show my work designing this interaction, here's a quick demo.

Despite the limited vocabulary, the interface was able to create a good looking website and the conversational aspect added to the experience. Small win? I'll take it.

PS. I really wanted to record the version where instead of typing these instructions, I could speak them out loud and the bot would reply back with its bot voice but I just couldn't manage a proper recording setup to make it look spectacular.

Future

What's next?

Even though there's so much more that can be done with this project, I'm quite happy with this initial version. I was able to create a completely functional website using this project which is a great measure of success for me. In the future, I will keep iterating the design and algorithm to improve the interactivity of this project and hopefully make this public so that people who are new to web development can use it to implement their designs.

Read Next

Workaholic