Personal note: It’s 1 AM. Many of my cool thoughts come and go between 11 PM - 3 AM. Today I stayed up to write it, lest it escape as well. If you have thoughts on anything written in this space, let me know! And yes, the title of this piece is inspired by the Brat deluxe featuring Caroline Polachek (official audio). Might be incoherent writing but I think the arguments are there.
I’ve been thinking about how we create frameworks to measure things we don’t fully understand yet.1
Measuring humans
In law, we say pure facts decide cases, but lawyers actually arrange those facts into compelling narratives because humans are terrible at separating logic from feeling. Yet some arguments, as determined by the judge, prove stronger than others. And if everyone were qualified on paper to do the job, then how does the interview assess for the human as a whole?
After my peers and I ran our Pacer tests in middle school, we would look back and reassure ourselves that our scores would have been higher, that we could have run for longer. Surely, I could have done one more lap back-and-forth (for simplicity, let us assume the young person’s ego is not involved here). But in the moment, the reason we chose to stop at our very moments was that we could not endure the pace any longer. Our legs were tired, we were thirsty for a drink, our lungs were on fire, and we could not keep up with the increasing frequency of the beep. Of course, our signal to the brain is to opt out of the Pacer test in the instant. In the moment, reality was different. But our retrospective selves, free from the experience of physical pain in real time, spoke differently.
The same pattern appears in human relationships and assessment. Did you really love them, if you speak negatively now? Does it negate the fact that your past self might have thought differently?
How much of this recency and post-event bias affects our communication, knowledge of past experiences, and even in predicting the future?
We are generally bad at decision-making and consistency, because we are human beings, not robots, as measured by accuracy or productivity. Thus, apart from the moment, I have nothing to judge but the fact that everyone is more-or-less trying and doing their best in each moment, given the information and capacity they have. This framework is my argument for empathy. This concept, interestingly, parallels the greedy algorithm: where the best decision is made by the fallible human at each moment, and not optimally (even if we deem it so, greedy is not always optimal). And, even in assuming a person’s good judgment, when one can, this is a strong reason to have a first-hand experience for oneself (e.g.: travel, interaction with others), rather than simply taking a third-person account at their word. Sometimes the conclusion may be the same, though in more nuanced settings, an experience may differ greatly.
Measuring intelligence and progress
To the judge of one’s technical ability or musical performance: is it relative, and perhaps even relative to one’s own? Speaking on my personal experience (teaching for a decade), when grading students on a scale, it helps to make rubrics as clear-cut and objective as possible, with little to no room for error. This may be more work upfront on the instructor’s part, yet worth it as it is more fair for the student. Without it, I witnessed a lack of imposed standards. To one, “good explanation, 10/10” would become a score, while to the other grader, they might have given the same answer by the same student: “lacking substance or value, 2/10”. We were grading a proof, which, to some standard, is supposedly grounded in a truth, correct or incorrect. All the more complex in the arts. And yet, as a musician and artist, I know there is beauty pointed towards objectivity, for I know what is beautiful and what is good, and what interpretation (or lack thereof) is not pleasing to the ear and soul.
Similarly to how we evaluate models (“evals”, something I worked on temporarily this past summer), we need benchmarks. In AI evaluation, we build benchmarks even though researchers still debate what we’re measuring toward - what counts as intelligence, and how do we know when we’ve reached the unknown unknown? The scientists (researchers, engineers, and research engineers) who have worked in the frontier space for any problem long enough can grasp clear ideas of what to do next, or what to hedge their bets for and against. We can develop benchmarks based on current knowns to extend to current unknowns. We use what we know to be “good” as a baseline, given what capabilities can currently be achieved, and make better guesses on where things are heading.
I believe such is the art of improvement, learning, and exploring the frontier of science and progress – to not fully know, but the goal in poking holes, having the world respond to our provocation, and understanding a little bit more of what is not yet and what could be. Thus is how I learn: when I learn what I do not know, I seek to explore and close the gap. There is much productive confusion and pain along the way, which I embrace with open arms.
This gap between our clean measurement systems and messy reality isn't necessarily a flaw. Had a piano been without the chromatic black notes between the standard ivory keys, music would be boring and colorless. Of course, the extra notes allow for more of the unpleasant, and the revealing the ugly. Yet it also creates dissonance and beauty. And we hear all the smooth sounds on a stringed instrument (portamento, like density between numbers in math, or mass in a volume in physics), so why did we stop at hearing 12 unique notes on a piano? Why is it difficult for the human brain visualize anything past the fourth dimension, yet understand and trust the understanding of the computer that performs such calculations? Perhaps our brains and hearts could not handle more; we have enough permutations and thoughts to sit with.
To this end, I believe every field is really doing the same thing through different lenses:
Everything is an art. I can admire the beauty, art, and emotion behind [X].
Everything is a law. I learn the laws, framing, theories, philosophies, and lemmas that build and shape [X].
Everything is a science. I experiment, study, discover, and understand [X] and how it works.
Everything is a language. I name and speak the tongue of the people who know and know about [X].
Everything is engineering. With [X] being an engineering problem, I can build or build towards it.
This seems like a strange loop where, from science to the arts, everything is interconnected. It was there all along, and we are just discovering it.
EDIT: There are also fields that I am not aware of, that I believe can be extended with this argument: Everything is biology. Everything is economics. Etc. I just learned about the Grand Unified Theory.
Also: not everything is meant to be measured. I am not advocating for this on large scale, though in terms of some benchmarks and applications.
“to not fully know, but the goal in poking holes, having the world respond to our provocation” good line