Shady Phoney

It's an especially weird time to write non-essential software, with climate change "knocking" at the door, the ongoing invasion of Ukraine by Russia, the horrific terrorist attack by Hamas, and the brutal military response by the Israeli government. In my country right-wing populism is also once again on the rise.
I am immensely grateful to people who use the power of their humanity, profession or finances to support actions that further human flourishing on these fronts or bring nuance to discussions that have lost it in the heat of battle (/social media).

Then there is also the whole AI boom, which fills me with both uncertainty and excitement, about how my craft will change. Still, this craft and the art within is what fills me with meaning and joy. A lot of my earliest memories of feeling inspired revolve around technology.

So, despite (to spite?) these larger forces, I have taken time from my time off to explore programming on mobile devices. I find it disappointing that the devices we carry with us at all times only allow us to consume the "finished software" without supporting us much in its creation. There would be a lot of hindrances to adopting a full-fledged programming environment. Programming tooling is as complex in its system requirements as it is broad in its feature set. Programmers value typing efficiency which phone keyboards will never give you.
Thus I started looking for a vertical slice that could work with less complexity and…

…I ended up building a prototype of a mobile (pixel) shader editor. Learning about shaders was on my shortlist for a while and this turned out to be a perfect candidate for an entirely integrated mobile solution.

Shader programming is a weird animal, and my head is still fighting to wrap itself around it. It asks you to have a mathematical sense of geometry and use its algebra to warp space. This is a stark contrast coming from 3d modeling tools, or game engines, where you "simply place objects in a scene" (with more direct manipulation around the corner).

Then there are artists like kishimisu who's work have this ethereal, kaleidoscopic, church window quality that tickle my inner mathematical realist (/esoteric numerologist?).

View this post on Instagram

A post shared by @kishimisu

Such intricacies probably cannot be created in my tool yet (though I honestly don't know about the exact limits). In the initial version of shd.is, a shader program is just a list of assignments, where the final value of the color variable (🎨) shades the current pixel. This program is run for each pixel et voila, you have a picture, by which I mean you have now learned the basics of pixel shaders.

This is what the starter program looks like:

First things first: Yes, variable names can contain emojis. You are seeing three built-in variables here:

• 🎨 - the aforementioned color of the current pixel, in RGB
• 📍 - the coordinate of a pixel, a 2d-vector
• 🔎 - the resolution of the canvas, a 2d-vector

That .xy1 is a shader programming technique called swizzling, an operation that transform a vector’s components. It is used here to transform these 2d vectors into 3d ones, with the z-dimension (the B in RGB here) set to 1. Thus the blue tint.

Let's look at the controls next, specifically selection. I took inspiration from the iOS keyboard and its spacebar, which already acts like touchpad, so I made it look like a touchpad (the big rect in the middle). It allows for smooth, precise control without leaving the touchy comfort of the lower bottom of the screen. This is what it looks like in action:

As you can see, you cannot only select existing elements, but also placeholders around an assignment. By selecting those and making an input, you can create new assignments at that position. Let’s use that to make… err… use time:

Okay wow, that was kind of a lot. Here are most of the steps:

1. Use touchpad to select the first placeholder
2. Tap variable button to create a variable named 👽
3. Swipe variable button to set the value to ⏳ (time)
4. Swipe function button to wrap time in a sine function, thus creating an oscillation around -1 and 1, driven by time
5. Use the touchpad to select the position and multiple it by 👽

There is a feature hiding in that last step: Tap the touchpad to extend the selection to the parent node.
When writing code on my laptop this is one of the most essential interactions, and also one an astounding number of coders don't know about.
Here are the docs for it in VSCode and in IntelliJ.

Building on that there is also an action called unwrap, which replaces a parent node with the selected child. This is what it looks like:

This action was born in my other structured editor projects and has also become essential for me. Check out my VSCode extension Soy if you want to use it in your computer programming (cross-promotion, baby).

There are more actions I could talk about, but I will keep it brisk for now. If you play with shd.is you will find it has the basics: clipboard, history and typical numerical actions.

Also: don't look at it on your computer, use your phone.

What it does not have is things a programmer might consider essential, namely loops, creating your own functions and conditionals, just to name a few. There is a (to me) surprising amount of shading you can do without those, but I am definitely missing loops (and thus scheming how to implement them).
I built the app in React Native, and I intend to release native versions of it to give users more screen space, faster shaders, and haptic feedback.

Thanks to my bestie Reuben who is always listening and engaging with my FoCing ramblings and has spent an amazing saturday with me working with an earlier prototype on his phone, while I live-fixed next to him. This is my ideal way of working and I wish I had it every day.