YAPL
I have observed the rise of a new language: YAPL, (recursive acronym for “YAPL Ain’t a Programming Language”, or “Yet Another Programming Language”, with pronunciation left to the imagination of the reader). I thought about expanding it as as “Y A Programming Language” (pronounced “Why A Programming Language?”), but refrained from it because it could be interpreted as “Y APL” which might lead people astray…. Anyways, anyone dealing with a CI/CD system, or any cloud based service for that matter, are likely to have bumped into some dialect of the language. Key features are
- YAML syntax (the common denominator between various dialects)
- Custom semantics (i.e. interpreting the YAML syntax differs between implementations)
- Support for expressions with a custom DSL 1 within a
${...}syntactic construct
In short, all YAPL derivatives share the same syntax but have different (but usually similar) semantics. For example, here is the code for a cleanup workflow in Google Cloud used in one of my projects
main:
params: []
steps:
- init:
assign:
- project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- job_location: europe-north1
- job_namespace: ${"namespaces/" + project_id + "/jobs/"}
- buckets:
- bucket-A
- bucket-B
- bucket-C
- cleanup-loop:
for:
value: bucket
in: ${buckets}
steps:
- cleanup-bucket:
call: googleapis.run.v1.namespaces.jobs.run
args:
name: ${job_namespace + "cleanup"}
location: ${job_location}
connector_params:
timeout: 900
body:
overrides:
timeoutSeconds: 900
containerOverrides:
args:
- ${bucket}Yep, that is how a for loop in YAPL looks like. Beautiful isn’t? In my opinion, turning a markup language (that is, a language used for representing data) into a scripting language is an anti-pattern. Besides turning a data language into something it isn’t, do people even like YAML to begin with? Seriously, I still can’t get the indentation right!
What alternatives are there, and why do people over and over reproduce this anti-pattern? To begin with, serverless workflows are functional. Functional programming is a trending pattern in software in general and several languages embrace it. Hence, a good replacement to YAPL should be a functional language.
For example, here is how it could look like in Clojure
(defn main []
(let [project-id (sys/get-env "GOOGLE_CLOUD_PROJECT_ID")
job-location "europe-north1"
job-namespace (str "namespaces/" project-id "/jobs/")
buckets ["bucket-A" "bucket-B" "bucket-C"]]
(doseq [bucket buckets]
(google-run-v1-jobs/run
:name (str job-namespace "cleanup")
:location job-location
:connector-params {:timeout 900}
{:overrides {:timeout-seconds 900
:container-overrides {:args [bucket]}}}))))
or Tcl
proc main {} {
set project_id ${::env}(GOOGLE_CLOUD_PROJECT_ID)
set job_location europe-north1
set job_namespace [concat namespaces/ ${project_id} /jobs/]
buckets {bucket-A bucket-B bucket-C}
foreach bucket $buckets {
google_run_v1_jobs::run \
-name [concat $job_namespace cleanup] \
-location $job_location \
-connector_params {timeout 900} {
overrides {
timeoutSeconds 900
containerOverrides {
args [list $bucket]
}
}
}
}
}
But to be realistic, this ain’t gonna happen. Adopting a non-popular (not the same as unpopular!) language is considered high risk in software business. Even more so by writing your own language. The big question mark is why not some existing and popular language has been adopted instead?
JSOAP
To answer that question, let us tweak the YAPL code snippet a bit. Bare with me, we will have to take one step back before we can move two forward. Note that YAML data (that is, excluding comments) is equivalent to JSON, that is, they represent the same ADT 2. Even though this is never an option in YAPL based systems, we can easily rewrite a piece of YAPL into a new, unholy form of JSON, which I will call JSOAP, acronym for “Java Script Objects Ain’t Programs”. I though about just calling it “JSONScript” first, but found it too laughable when expanded…
{
"main": {
"params": [],
"steps": [
{
"init": {
"assign": [
{"project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"},
{"job_location": "europe-north1"},
{"job_namespace": "${\"namespace/\" + project_id + \"/jobs/\"}"},
{"buckets": ["bucket-A", "bucket-B", "bucket-C"]}
]
},
"cleanupLoop": {
"for": {
"value": "bucket",
"in": "${buckets}",
"steps": [
{
"cleanupBucket": {
"call": "googleapis.run.v1.namespaces.jobs.run",
"args": {
"name": "${job_namespace + \"cleanup\"}",
"location": "${job_location}",
"connector_params": {
"timeout": 900
},
"body": {
"overrides": {
"timeoutSeconds": 900,
"containerOverrides": {
"args": ["${bucket}"]
}
}
}
}
}
}
]
}
}
}
]
}
}
Now that is straight out horrendous! But remember that this is equivalent to the original YAML snippet.
The escaping of the quotes becomes ugly in JSOAP since strings are always quoted (i.e. the expression ${...} is a string in YAML, but not in JSON unless quoted as "${...}").
You also have to escape nested strings (such as "\"namespace/\" + ..."), making it even less readable.
What if we replace all occurrences of "${...}" with `${...}`? Then we get back interpolated strings from JavaScript!
So, by extending JSON with string interpolation, we are actually just performing ordinary object construction in JS.
Let us rewrite the JSOAP snippet in JS by removing unnecessary quotes, replacing occurrences of `${x}` with x,
DSL library functions with JS standard library functions, and some idiomatic string interpolation. Then we end up with this
{
main: {
params: [],
steps: [
{
init: {
assign: [
{project_id: process.env.GOOGLE_CLOUD_PROJECT_ID},
{job_location: "europe-north1"},
{job_namespace: `namespaces/${project_id}/jobs`},
{buckets: ["bucket-A", "bucket-B", "bucket-C"]}
]
},
cleanupLoop: {
for: {
value: "bucket",
in: buckets,
steps: [
{
cleanupBucket: {
call: "googleapis.run.v1.namespaces.jobs.run",
args: {
name: `${job_namespace}/cleanup`,
location: job_location,
connector_params: {
timeout: 900
},
body: {
overrides: {
timeoutSeconds: 900,
containerOverrides: {
args: [bucket]
}
}
}
}
}
}
]
}
}
}
]
}
}Looks a bit more clean now when written in JS and not JSON (I have never understood why JSON syntax differs from the syntax of JavaScript Objects actually). Point of this little exercise is that now when we use a proper scripting language to express our intents, why not take the next step and rewrite it with Plain Old Functions (POF!)
function main() {
const project_id = process.env.GOOGLE_CLOUD_PROJECT_ID;
const job_location = "europe-north1";
const job_namespace = "namespace/" + project_id + "/jobs/";
const buckets = ["bucket-A", "bucket-B", "bucket-C"];
for (let value of buckets) {
googleapis.run.v1.namespaces.jobs.run({
name: job_namespace + "cleanup",
location: job_location,
connector_params: {
timeout: 900
},
body: {
overrides: {
timeoutSeconds: 900,
containerOverrides: {
args: [bucket]
}
}
}
});
}
}It is still not as concise as the snippets written in Clojure and Tcl, but I would still prefer this every time over the YAPL/JSOAP mess.
Related posts
-
(Jun 21, 2024): The configuration complexity clock: Spot on post on the topic of configuration complexity in general, and an observation of the recurring anti-pattern of trying to solve it with advanced methods.
-
(Jun 21, 2024): Why the fuck are we templating yaml?: Yet another rant on YAML. Recognizes YAML as a bad format for automatically generated configurations (partly due to spaces being part of the syntax). Promotes jsonnet, which seems tempting at first, but then you realize that this is just “9 o’clock” on the configuration complexity clock. Back to basics, use JS instead is still my conclusion.
-
(Jun 21, 2024): INTERCAL, YAML, and other horrible programming languages: Similar rant to this post with some historical notes on XML and the esoteric INTERCAL language. To quote: “Writing control flow in a config file is like hammering in a screw. It’s a useful tool being used for the wrong job”.
-
(Jun 21, 2024): The inner platform effect: Gives a name to the anti-pattern of creating a configuration system equally or even more complicated than the original system. Not sure how applicable this is to configuration of cloud workflows since I do not have access to the source code of the original system, but I imagine how the YAML is parsed by Google in some normal language like JS or Go, which could have been exposed directly to the end user instead.
-
(Jun 21, 2024): Stages of despair in configuration: Accurately reflects my feelings as my cloud workflow complexity grows out of control (remember that the snippet I am using in this post is made from the smallest workflow in my app!).
Errata
-
(Jun 21, 2024): The Tcl snippet has an error
${::env}(GOOGLE_CLOUD_PROJECT_ID), which should be$::env(GOOGLE_CLOUD_PROJECT_ID). It was left in due to some desperate attempts to fix a formatting issue related backslash escaping by the Jekyll HTML/Liquid processor. I suspected a conflicting interaction with MathJax processing of$-signs, but it was all resolved by replacing{% highlight tcl %}with triple back-quotes (not sure how to escape that out in Kramdown even!). I even had to escape this sentence itself with raw/endraw blocks to render it properly, escaping can be really tricky! As I dig deeper into the Liquid DSL used by Jekyll I realize that this is on the edge of being Yet Another Programming Language example by itself (I would say approximately 7 o’clock on the “markdown complexity clock”). -
(Jun 21, 2024): The JSOAP snippet rewritten using JS notation would never work since variable lookup such as the reference to
job_locationwould never be resolved. It’s just a really bad idea.