notgull

SCP-093 is a Timeless Masterpiece

2023-12-23T00:00:00+00:00

SCP-093 is an absolute masterwork. Here’s why.

This essay is adapted from this comment on the SCP-093 discussion page, by me, in 2019. So, the audience of the post of more “people who are familiar with the SCP wiki” than the usual audience of this blog.

Some time ago, I was trying to write a long, exploration-log based SCP. However, I found myself struggling with keeping pace with the logs over a long period of time. So I decided to consult one of my favorite skips: SCP-093.

I consider SCP-093 to be the second best skip of all time, second only to SCP-2000, which probably deserves its own essay. I wanted to go in depth and figure out what made 093 different from all of the other 4000+ skips that have shown up since its writing in early 2009.

Context

SCP-093 is about a disc that showed up on the Red Sea. The disc acts as a portal to another world, where everybody’s dead and monsters roam the Earth. The SCP Foundation eventually discovers that this was the work of a singular entity, “Him”, who convinced the world it was their god.

Out of universe, SCP-093 used to just be a disc that glowed when people held it. The entry dated back to the EditThis wiki, before Wikidot, and its original author remains anonymous to this day. Eventually, an enterprising user named NekoChris completely rewrote SCP-093, adding a mechanic where it rolled towards mirrors, as well as the new test logs. Aside from some edits made by other users for grammar and tone, this is the version we’re reading today.

This section is probably unnecessary, but I feel like it’s important to understand these things in the context of their creation. Without further ado, let’s move onto the skip itself.

SCProcedures

I wanted to do a line-by-line analysis of this. So let’s skip the ubiquitous Item # and Object Class and skip to the Special Containment Procedures.

Special Containment Procedures: See testing document SCP-093-T1 for outline of testing conditions. SCP-093 must remain on a mirror at all times and under video surveillance. Admittance into the area of SCP-093’s containment must be authorized only with proper video recording and subject retrieval procedures in place. Any attempt to use SCP-093 outside of an approved test will be dealt with severely, up to and including termination.

Short and to the point. Keep it on a mirror, make sure it’s under surveillance, and don’t let anyone in without approval. This foreshadows what’s coming ahead, which is what any good SCProcedures should do.

There are, however, some points here I’d like to point out.

Any attempt to use SCP-093 outside of an approved test will be dealt with severely, up to and including termination.

This is kinda a recurring thing that I dislike about older SCPs. The SCP Foundation hires highly trained, valuable scientists and agents, so why would they just kill them? I think “termination of employment” would work better here.

See testing document SCP-093-T1 for outline of testing conditions.

I had to take a moment and think about not only why this line was in the SCProcedures, but why it was the first thing that it said in there. If this SCP-093 document was thrust into the hands of a containment agent, why would they want them to read “look at this document for testing conditions” first?

I don’t know, but I have a few ideas:

NekoChris wanted to emphasize that this is a testing log skip.
In universe, the testing procedures are really important to make sure He doesn’t breach containment (more on Him later)
My personal favorite: notice that it says “testing document T1” (where the mirror testing is) rather than “testing document T2” (where the exploring is). I think that this is outlined at the top of the file because they want to make sure people are testing 093 with the mirrors correctly, so they don’t accidentally enter the alternate reality. This foreshadows that the Foundation doesn’t want people entering the alternate reality, even by accident. I’ll bring this up later in this essay.

Description

Description: SCP-093 is a primarily red disc carved from a stone composite resembling cinnabar, with circular engravings and unknown symbols carved at 0.5 cm depth around the entire object. Deeper cuts are present on SCP-093 with a depth of 1 to 1.5 cm. SCP-093 is 7.62 cm in diameter and fits comfortably into most palms without abrasion. SCP-093 will change hue when held by a living individual. The colors taken by SCP-093 are still being researched to establish a link. Current belief holds that the changes depend upon regrets carried by the holder.

This is fairly straightforwards. The first paragraph describes the appearance of the object, that it can be held in your hand without issue, and that it changes color when it’s held by different people. Nothing big.

By the way, remember how I said that this was a rewrite of a previous low-quality skip? Well, NekoChris pretty much summed up that entire skip in a single paragraph. Which is, number one, an amazing feat of writing and, number two, I feel is done intentionally. He’s getting the old skip out of the way, so he can focus on the new stuff.

If SCP-093 is removed from a mirror and not held by a person, it will seek out the nearest mirror-like surface. SCP-093 has been observed to travel in the largest possible circle while rolling, building up phenomenal speed.

Again, pretty straightforwards. 093 wants to be in front of a mirror, so it’ll roll towards the mirror and break any obstacle in its way to do so.

Additional Notes: No records exist to clarify the nature of SCP-093’s discovery or presence in the Foundation. See SCP-093-OD. Since no records exist explaining SCP-093’s method of containment, a test procedure was initiated to establish why mirrors must be used to contain it. The results of SCP-093-T1 lead to the discovery of living beings holding SCP-093 being able to move through mirrors and the series of tests in SCP-093-T2 to ascertain the destination reached through this travel.

I’ll come back to this in a minute, but they’ve lost the records for SCP-093, so they have to do some tests to figure out what it does. This leads into the mirror tests, which lead into the exploration logs.

Original Documentation

Before we get to the tests, we get to see the original documentation of SCP-093. This is done in a very interesting way; the SCP-093-OD is actually the pre-rewrite version of SCP-093. This is interesting because, as far as I know, no other rewrites include their pre-rewrite versions wholesale.

However, the way that it’s included, it’s not just a callback to the original. NekoChris used it in such a way that, just by being there, it progresses the story. Let’s look at two points in this section that I feel are important:

SCP-093 resembled the documented blue for 54:34 at 1:23 on 26 April 1986 coincidentally when the body of 194-9834 was discovered in Research Facility █████.

Ties between 194-9834 and SCP-093 remain inconclusive and effects of prolonged exposure to 093 remain unknown except for infrequent reports of periods of calmness and in the case of 242-0049 as periodic waves of depression, loss of balance and thoughts of suicide.

Okay, in this, it asserts that SCP-093 causes calmness and, in some cases, suicidal thoughts. Remember that this is never mentioned for the rest of the skip. I think that there are two options here:

NekoChris either accidentally forgot to make a callback to this property, or deliberately did not call back to it to avoid mucking with his narrative.
In between 1985 and whatever date the current document takes place at, SCP-093 somehow lost those anomalous properties.

Additional Notes: Origins of 093 remain unknown and documents of recovery of 093 have since been destroyed in a fire in Research Facility █████, 09 December 1989.

At first glance, this may seem inconsequential. But, remember this line:

Additional Notes: No records exist to clarify the nature of SCP-093’s discovery or presence in the Foundation. See SCP-093-OD. Since no records exist explaining SCP-093’s method of containment, a test procedure was initiated to establish why mirrors must be used to contain it.

You see, these two relatively small blocks of text, written by two different authors, years apart from each other, instantly create a storyline involving the Foundation in less words and in a much more compelling way than most skips do today.

Let me explain; according to the “Additional Notes” section of the old documentation, there was a fire that destroyed most of the documentation for SCP-093, leaving them not knowing why they contained it on a mirror, or what the colors of SCP-093 meant. First off, this is actually kind of meta; it explains why a lot of Series I skips (especially the older ones) are so short: a fire destroyed their documentation.

However, NekoChris took advantage of this line in his rewrite by saying that the Foundation didn’t know about why it was laying on a mirror, and that testing was beginning to find out why.

This paints the following picture for me: a researcher, or bureaucrat, or whatever, is going through the old Foundation files, when they find the file for SCP-093. From reading it, they find out that we’ve had some red mood-ring disc lying on a (very expensive) mirror since 1968 for seemingly no rhyme or reason. Of course, they say something along the lines of “why?” followed by “we need some tests.”

This not only establishes the beginning of a narrative in only a few words, but sets up something grand that I don’t see in a lot of skips, new or old, do: it sets up the Foundation as a main character, and sets it up for characterization.

What I mean by this: in most skips, you learn information from a certain point of view, whether that of a researcher, the SCP itself, an MTF team, or just none at all. However, in SCP-093, the Foundation itself takes this role. The reader learns new information at the rate at which the Foundation discovers it, and almost gets to feel the role of the Foundation as they figure out the anomalous properties of SCP-093 and explore the world beyond. In addition, this also allows for some pretty cool characterization of the Foundation; we see the Foundation’s motive change from simple testing, to exploration, to full-on planetary defense. I’m not saying that 093 is the only skip to do this; many other skips characterize the Foundation’s goals, history, and parts as well. But they often use paragraphs upon paragraphs to do it, while 093 does it and such a wonderfully subtle way that it’s hardly noticeable without paying attention.

Just a reminder, all of this is set up in just around five lines. Five. Lines. This is economy of language at its finest, and is really genius level writing.

Anyways, now that the main character is set and the plot is ready to go, let’s start with the part where SCP-093 really gets interesting.

Test Log 1

Mirrored surface, brass frame, retail-grade mirror: SCP-093 rests without activity when placed on the mirror. This test alone removes the need for costly silver or wooden containment systems.

This is actually kind of funny, because it feels like something of a roast directed at the original author.

This test log really just restates what we’ve already learned from the Description, and principally serves to build up to the final log:

A person holding SCP-093 placing it on a mirror: This test was accidental, the result of one of the staff tripping another after some debate about who would be covering the lunch tab. As a result of the behavior of the researchers, it was discovered that a person holding SCP-093 and placing it against a mirror will in fact move into the mirror.

First, a complaint: along with the “termination punishment” thing mentioned above, this is how you can tell that this was written in early 2009; the era of lolFoundation, Keter Duty, and Omega-7. Why would a researcher bring SCP-093, a dangerous object that can punch through walls and has been known to cause suicides, to lunch? Why would they have it out while walking? Why is there a mirror in the hallway? Why would a researcher trip another researcher while he was carrying aforementioned object that punches through walls while it’s not in the palm of someone’s hand?

To be honest, I headcanon this as some kind of cover-up for some O5-eyes-only secret. Although I do admit that this line has personality, I think a simple “a D-class tripped and accidentally fell through a mirror” would work as well.

Alright, rant over. What this does well: it kickstarts the upcoming plot. We already knew that 093 rolls towards mirrors, but nobody knew why 093 rolls towards mirrors. In this, it’s implied that the whole reason why 093 rolls towards mirrors is to show us to this alternate realm. Reminds me of that one movie where there’s that bird that shows the main character to the gateway to the alternate dimensions where she’s supposed to go.

Now we know that SCP-093 + Mirror = Gateway to an Alternate Dimension. And now our main character, the Foundation, turns its motive from containment to exploration.

Test Log 2

(By the way, I’d like to interrupt this by saying that I really like this table-header format. It’s sad that, as far as I know, there aren’t any skips outside of 093 that use it.)

We transition here into the exploration logs, which is where the meat of this skip is. We get an idea of what our subjects are taking into the alternate dimension, and then we’re thrown into the Blue Test.

Blue Test

Camera activates, flickers to view. Subject is looking out over the same field reported by technicians. Looks like typical lowland plains, everything has a heavy blue tinge overlapping the normal colors. No discernible landmarks visible as subject pans view left to right, only grass, weeds, and a breeze moving the taller grass. No trees. No living beings visible.

The first paragraph asserts that we’re in some kind of plains. However, there are no trees, and no living beings. Right off the bat, it is asserted that we’re either in some kind of remote part of this alternate universe, or that everything is dead. This is a post-apocalyptic universe.

Eventually, the subject enters a hole in the ground, which leads to a tunnel, which leads to some kind of bunker.

Room is bare, no contents, but walls are filthy. Subject states material on walls isn’t dirt, but he can’t identify it, seems to resemble melted plastic but is brown in color rather than black.

This is where we’re first introduced to what I’ll be calling “mystery goo”. It smells bad, and it’s everywhere. It serves as sort of a “recurring mystery”, where it appears everywhere, and we’re slowly clued into its origin.

Walls of room are clean as is floor, ceiling is coated in the same strange brown material as the third room. In this room there is a makeshift cot made from aged blankets with a pillow, a wooden crate containing open boxes of what appears to have been food stuffs, language appears on video as squiggles however subject states they simply read ‘Cereal’. A second crate in the room contains what appear to be empty water bottles that have dried out. A book lays next to the cot, closed, no title or identifying marks.

Okay, this is definitely a survival bunker. This confirms the belief that this is some kind of post-apocalypse, and that there used to be people here. But now, there is just the weird mystery goo, collected on the ceiling.

Careful review of the following ten seconds of tape shows that as the camera pans, a figure is visible at the end of the tunnel where the seventh door is. The door is open only enough for a face to be seen through a crack just before the door silently closes. No details can be seen.

This is another one of those “recurring mysteries” to help keep readers hooked: these mysterious shadow people that show up everywhere. After all, it’s made clear that this is some kind of post-apocalypse, so why are there people?

This is almost glossed over- which is good. It’s kind of a “wait, what the hell?” moment, and the fact that it’s almost entirely unmentioned after the fact solidifies that moment.

Then there’s some cable trouble, then the subject gets pulled out of the hole by some unseen force, and the camera sees 37 creepy ghost people. They’re unidentifiable, and they’re watching the subject for some reason.

Control requests subject return following cable path and screams are caught on the audio with panic from subject. Five shots fired as subject aims pistol at something not visible on camera. Control reports being able to see subject returning toward point of origin while camera shows wire disappearing into a point floating in the air. As subject passes this point all cable is now in the pulley system and camera films only the floor. Control reports that the mirror took approximately five seconds to return to a reflection and SCP-093 remained blue in color until one hour after being recovered from subject.

A vile smelling fluid was present on subject’s clothes around his hands when firearm was recovered. This fluid dried quickly and was deemed insignificant of study due to lack of quality sample. Control personnel monitoring the mirror state having seen a massive human being, crawling on the ground, easily fifty times the size of a normal person with no facial features and a very short arm reach, pulling itself toward the mirror before it returned to a reflection. Due to proximity fine details could not be made out but at least one observer noted the being appeared to have been shot from the marks in the otherwise smooth featureless face.

This is where we get a look at the “antagonist” of SCP-093, for lack of a better word: the Unclean. In the first test log, the reader only gets a glimpse of them; they’re big, they’re scary, and they’re chasing the Subject.

Field Test Kit recovered from subject containing a news paper article that reads: [DATA EXPUNGED] and was filed as item [DATA EXPUNGED].

These expungements may seem unnecessary at first glance; however, this is some story information that is best saved ‘til later, and (IMO) actually convinces the reader to read on out of curiosity as to what’s behind the expungements.

All in all, a pretty good opening log for this skip. It introduces the parallel universe, the fact that it’s a post-apocalypse, the mystery goo, the ghost people, and the Unclean in just the right amount for it to be spooky, but also to leave the reader wanting more. I do think it could be improved by adding some foreshadowing to the religious apocalypse (I.e. maybe a cross in one of the bunker rooms?) and I think that the Unclean were described a bit too much for what should’ve been a vague hint at a monster (I.e. instead, maybe say “technicians saw a glimpse of a large humanoid figure crawling on its hand before the mirror returned to a true reflection” and leave it at that), but for what it is, it excellently introduces the concepts and players which are built upon in the remaining logs.

Green Test

No landmarks from Test 1 are discernable as subject pans camera over area.

Present is a field, long abandoned, in the middle of which stands the remains of a scarecrow of unknown design, fragments left are rotted and torn. Nothing grows in the tilled land. A farm house is visible to the right of the field, large, two stories, a basement shelter entrance is visible at one end.

Alright, the first few lines tell the audience the following:

This is in a different place than Blue Test.
We are on a farm.
Everything is dead.
There’s shelters everywhere, which implies that people knew about whatever apocalypse happened here beforehand.

As subject pans the area a metal hatch is visible in the ground, similar to a bulkhead on a submarine with a turn handle. Subject remarks that the smell is at its worst around the hatch and the dirt around the hatch is noted as being clumped and claylike.

This is repeating the whole “stench” and “mystery goo” thing from Blue Test. What this does is make the audience anticipate the inevitable “mystery goo” and perhaps even the ghost people from Blue Test.

On the beds are two skeletons and on the floor is a third, lying next to which is a simple six shooter revolver containing no ammunition. Three spent casings are across the floor near the gun.

I haven’t been commenting a lot for a reason I’ll bring up later, but this gives the audience new information: there was a struggle. They didn’t just keel over, they were killed by something.

In the distance, approximately 700 m from the farm, two massive, humanoid beings are crawling across the landscape.

This further introduces us to the Unclean, which were only slightly glimpsed in the last test.

There is a recliner, a couch, and a television all of 1950s style design. In the recliner is a laptop whose case also resembles 1950s decor and is coated in heavy dust.

This is some further world building. When this civilization ended, it was in the 50’s, or some analogous point. This helps to further paint a picture of this world in the reader’s mind.

To conclude, at first glance, this seems to be a retread of the first log. In fact, the reason why I didn’t do as many quotes was because a lot of the structure is almost identical to the first. However, along with some decent world building, it gives us new information. Whatever apocalypse happened here wasn’t instant. There was a struggle, against something. Something that managed to get into the bunkers and kill an entire population. And if you’re like me, you’re beginning to suspect that it has something to do with the large human like things that we’ve seen twice now.

This all helps to build the mystery, along with the creep factor NekoChris has been setting up: the eerie feel of an empty world, roamed by these eldritch horrors.

Now onto the third test…

Violet Test

Subject is in what appears to be a modern downtown district similar to a city like New York. The streets are mostly bare except for a few cars of unknown make or model. These cars look highly advanced and streamlined.

Instead of another rural area like the last two tests, we instead start with a city area, which, if we had any doubts about whether or not this was an apocalypse or just a bunch of people leaving the countryside, this silenced those doubts.

In addition, this gives us a glimpse into this civilization’s tech level with the advanced cars, which we haven’t yet seen. This is some further world building.

Subject attempts to look into the car windows without being instructed to but backs away remarking there is a ‘rank ass stank’ coming from the areas around most of them.

Subject is persuaded to move closer to one car and does so with coughing, wiping off a window which is covered in dirt. The inside of the car appears to be completely filled with a strange brown matter, there is nothing at all visible other than the brown matter.

Control debates this issue while subject stares around the cityscape from the car. During one pan a face is clearly seen staring into the car, eyes watching the subject; however, this was not noticed until post-test footage review.

Now we also have the stench, the mystery goo, and the ghosts. These are familiar elements, and they’re all gotten out of the way before we really start the exploration.

A team of four armed personnel is sent through the mirror and proceeds to subject’s location.

This time around, we have an entire team of soldiers traveling with the subject. I feel like this is done to change things up from the last log.

The view of the city is astonishing. This building is one of the tallest visible but certainly not alone in its stature. The city below is gray and silent, no evidence of life at this altitude. Some buildings in the city have a strange brown growth that appears to have been splashed against them as if a gelatinous mass was flung and then seeped down before hardening.

Again, some more world building. They have some pretty good tech, but whatever they did they still weren’t able to fend off the apocalypse. “Gray and silent” in particular get to me in emphasizing that this is a ghost town, minus the Unclean.

On the desk is a notepad titled ‘From the desk of Dr. Borisizki, Blessed Purificationist’.

Yeah, this was implied with the whole “FaithfulOS” thing earlier, but this tells us that this world has something to do with the wide adoption and fundamentalism of some religion. This is further asserted by the test tubes.

The camera pans to a section of raised expressway across which one of the large torsos is crawling slowly. The entity turns its featureless head to look at the escort team, raises its head to the sky, and emits a bellowing sound.

This is an “oh, shit” moment. So far, all we’ve seen involving these entities is them either walking around, or vague glimpses of them. Now, we’re fighting one head on.

When a matching color is displayed and applied to the mirror the video receiver is visible and all individuals report a horrific smell.

Later review of the recovered camera shows escort member ██████ grasping at the air where entry point should be and then turning to look up at the oversized torso. A brown gel seems to drip off the creature as it moves that disappears shortly after being dislodged as if evaporating. Several shots are fired at the creature’s face with the automatic weapon carried by ██████ that land in the ‘face’ of the creature, causing a spray of less viscous brown liquid to pour forth from the ‘wounds’.

I’ll comment on these two parts later, but this reveals two things:

The stench, and therefore, the mystery goo, are both associated with the Unclean.
Since they’re associated, and given the goo and the corpses in the bunker, the Unclean were the cause of this apocalypse.

Also, a side note: I think NekoChris went out of his way here to not say “SCP-093-1” and instead use descriptions to describe the Unclean. I think this is to evoke more imagery with the techniques that they have.

This test actually reveals quite a bit about the world SCP-093 leads to. We see the high tech cities, the people in the vials, the religious connections, where the mystery goo comes from, and we experience the first real encounter with the Unclean. While Blue Test and Green Test paint a picture of a post-apocalyptic world ravaged by the Unclean, the Violet Test paints a picture of the world as it was before apocalypse. As much as I do detest the fact that the religious aspect is hardly touched upon in the first two logs, I feel like it makes the payoff in this log much more satisfying.

Yellow Test

D-class subjects no longer authorized for testing. Testing focus has been shifted to data collection after analyzing the articles brought back from the previous three tests to better understand the fate of the world accessed by SCP-093 and determine if safeguards or practices are required for our own world.

Can I just remark on how epic this line is? After painting a good enough picture of the world in 093, the Foundation shifts its focus from exploration to full-on planetary defense. Holy crap.

Just a reminder that this line comes after the test log where we, the reader, receive a definite linking between the apocalypse and the Unclean. Remember what I said earlier about the first parts of the skip setting up the Foundation as a “perspective character,” a character where we learn information at its pace? This is that setup in action, and it puts the audience is the Foundation’s shoes as it calls for planetary defense. This is one of those genius writing moments that make me love this skip.

The descent down the elevator is long, consuming 15 minutes, during this time the camera experiences one malfunction where the image jerks and turns to snow, restoring to show 14 other figures in the elevator with Dr. █████ as video pans around, all of whom move as he moves to allow him space. They remain for 35 seconds then the camera flickers to snow and returns, Dr. █████ is now alone in the elevator dancing as is assumed by the ducks and sways of the video feed.

Just wanted to remark about how imagery-inspiring this is. For all the complaints about the tone of these logs, they have some pretty good imagery and characterization.

The third view is facing the opposite direction as a camera glides vertically checking each observation station. A total of 10 can be counted and Dr. █████ is visible as the camera passes by his own station. Looking up, a hovering camera unit with no visible means of propulsion glides up past him. The fourth view shows the ground floor below the observation deck where a single astonishingly large torso being is crawling in circles, bumping into walls and changing directions. From the camera feed the creature’s estimated size is six stories.

This is actually setting up for the conclusion, and it does so in an unusual way: contradicting itself. NekoChris has spent the past three logs asserting that everything is dead, and the world has fallen apart. Now, we’re seeing rows upon rows of function PCs, and the containment of one of the Unclean. I’ll comment on this more later.

Analysis of ███-███ and the ammunition for it postponed for reason that it would require deconstruction of one of the rounds and they may be beneficial until testing of SCP-093 is resolved. Video ends.

Two things about this line:

███-███ is just a poor man’s SCP.-███
“they may be beneficial until testing of SCP-093 is resolved.” This is one of those sentences that really pop out at me. Remember how it was asserted earlier that the Foundation’s now focusing on defending the world from whatever happened in the 093-world? Yeah, they’re taking every weapon they can get to make sure they’re defensive.

All in all, the yellow log is really a continuation of the violet log, and helps to set up for the finale with its tonal shifts and feats of working technology.

Without further ado, let’s get to that finale.

Red Test

Service Technician ██████ was able to cause SCP-093 take on a fierce red hue and glow, much brighter than the object’s normal color.

This is one of those moments where you, the reader, know that shit’s about to go down. It’s when your heart starts beating, and you end up glued to the chair as you scroll downwards. This almost perfect buildup for what’s about to happen.

Video flickers to life and Technician ██████, known hereafter as Subject, is viewing a large cylindrical pillar that is rotating on its own. Object is of unknown height and appears to be 1.8 m (6 ft) in width. Holes are distributed throughout the object at seemingly random intervals. On occasion a beam of white light is emitted from these holes. Turning of the camera finds that the beams are connected to a multitude of objects similar to SCP-093 that are part of the room’s wall. The room turns out to also be cylindrical in shape with countless copies of SCP-093.

This makes two assertions. The first is more obvious: SCP-093 is a dimension-travel technology. However, the second, the fact that this is some kind of hub for SCP-093, doesn’t guarantee that we’re in the same universe.

Let me explain this theory of mine. For the past four tests, we’ve been thrust into an alternate dimension, which appears to be SCP-093’s intended destination. Now, we’re in some kind of 093 hub. Maybe the service technician somehow activated a “return to sender” feature on SCP-093 that brought him back to the source? I’ll discuss this one more later, once we’ve finished the SCP and we have all of the facts in one basket.

The ladder exits into a large clean room full of computer equipment that appears antiquated compared to previously encountered equipment. Large computers running on reel-to-reels are clicking and spinning at various locations, a light bulb of unknown meaning turns on for ten seconds then turns off. A large CRT monitor is displaying single words in 8 colors at roughly 5 second intervals. While observed the words ‘Clean’ ‘Unclean’ ‘Clean’ ‘Clean’ ‘Lost’ ‘Unclean’ flash on the screen.

Some more evidence to my “not the same dimension” theory: it’s obviously not the same level of technology. I theorize that the meaning of the words on the screen are referring to different universes accessible through the 093’s we saw in the hub. Whether they’re not infected by the Unclean, are infected, or if their SCP-093 instance is broken or lost.

All further SCP-093 tests have been discontinued while review of materials recovered is in effect.

I’ll comment on this line later.

All in all, the red test is a pretty good finale. It takes an “oh, crap” moment and extends it throughout the entire log. Now, let’s move on to the last log.

Recovered Materials

Any employee reading past this point who does not have proper classification should consider themselves to be terminated from employment and now subject to disciplinary actions up to and including: Forced administration of Class A Amnesic, immediate transfer to Keter class security, and death.

Another one of those “Series-1-esque” bits I talked about earlier. It’s charming, but I feel like punishing employees with death is the wrong way to go.

Most Holy Father Announces Progress, Unclean Being Cleansed!

Alright, this segment, I feel, is the result of a “Catch-22.” NekoChris couldn’t put it explicitly in the Blue Test, as it reveals lore that should be saved for later. But that means that they had to put it here. At this point, the payoff is good, but could be done in another way, I feel.

But if you see this symbol, if you see it.. you run boy, you run fast, you run far, and you hide, and you never go back where you saw it. That’s all I know. - I remember the symbol, was on the rock he kept on his neck under his shirt. Next day, pap was gone, nowhere to be found, dad weren’t sad, said he knew it’d happen one day, pap went home. See you soon dad, pap..

[DATA EXPUNGED] Symbol matched symbol found on SCP-093’s surface as one of the deeper engravings. Also matches symbols noticed on video feed of final test on SCP-093 duplicates.

This is some creepy, good shit.

Also, why would the symbol be expunged, especially in a file where all of the recovered, unexpunged information is? Given what we learn later about the origin of SCP-093, I’d guess that the symbol is this.

During preparation for the Tears subject went into a rage and the attending Hand went to recover a sedative. Jennifer tore her clothes off and screamed impure words at me so I locked the door and instructed the Hand to wait outside. I am half shameful to admit I laid with Jennifer a total of seven times before putting her to the Tears.

This line introduces us to the “Tears,” as well as emphasizes the corruption of whatever system was in place before the apocalypse.

The Eyes have dated the sample, it is older than myself, older than my elders. It is over 200 cycles in ages. 200!

I do not think this Unclean is alone. I have seen how they can get into places, between places. Between places! Is that where they have been, all this time? Between places?

The smell, it is so strong, it comes from all directions. It is not a smell of the dead, it is a smell that comes from something that should be dead but does not know how to die

This is some Kalinin-tier prose, and the note it’s in is great. It shows the perspective of the corruption and the Unclean from somebody who’s not on the top, but knows quite a bit. It’s great insight into this world, and it does it without being too over-explaining.

In the event of any Emergency requiring the Facility to be evacuated, all Clear-4 staff should report to Train Station 3 and use their Vial to call the Evacuation Train. Only one Vial is required to call the train and may contain any amount of Tears. An Empty Vial will not call the train. Clear 2 and 1 staff should remain at their posts until either 10 minutes after the departure of Clear-4 persons or until authorized by Clear-4 staff. Clear-3 staff should utilize the Protective Garments at their stations and weapon lockers before proceeding to designated Crisis Areas as dictated by Clear-4 staff.

This makes it clear that this is some 093-infected Foundation analogue. But remember the multidimensional hub? Yeah, NekoChris invented the multidimensional, all-intrusive Foundation before Scantron codified the concept in their 001 proposal.

I actually found a scientific report written by someone who stumbled here with a SCP-093 copy. These creatures are the result of exposure to a very pure form of His Tears resulting in a genetic apocalypse occurring within the exposed.

“Genetic apocalypse?”
This is where it all starts coming together for the reader. And it does so beautifully.

I think someone is in this facility, or someones, I keep hearing voices and requests coming from areas under the floor. I want to leave this before I explore the facility any further. I have sent SCP-093 back through the entry mirror to seal that gate. These things can’t be let into our world nor should we have anything to do with this one, we’re simply not smart enough to understand it all I feel.

I’ll comment on this more later, but this is one of those moments where it calls back to something earlier: specifically, the “bad grammar” entry just before this one. And something happened to that person.

They’re.. in my head I think.. I didn’t notice it till just now but, equipment in this room is starting to react to me, words on the screen, begging for help.

I’ve seen the faces, of the people, the Unclean. They show up on the pictures cast by the machine, in the room with me, watching me. I think, they’re everywhere on this world, only seen by machines now.

And it also explains the ghost people. Nice.

they showed me things when i touched them and its not quite like the records say. the unclean remember it all, every person they touch becomes part of them, safe inside them, but dead to us. every mind, every feeling, every terror, its eternal to them. i kind of want to join them but.. too much to do.. they want me to.. find him, kill him.

Oh no.

there was no war it was him him him him him IT. IT. it came from between the folds of time and space and worlds and light and dark something that is but should not be slipped in and called out to them as their god and they believed it and they tasted it and touched it and layed with it and became its property and did its will and IT IS STILL HERE the scp-093 it brought with it pulled forcefully with it built it i don’t know they don’t know but it belongs to him it lets him move between places between worlds so i BROKE IT ha ha ha i threw pieces of it away and through holes so those doors are closed just like ours is closed and i can’t go home so what else can i do

Oh no.

it calls out through the rock, somehow, it knows where they are but can’t touch them, but if you hide the rock he can’t call out and he’s stuck too i got you you son of a bitch I GOT YOU BANG BANG ha ha

i touched him. with my fist. and my gun. and he fell down. but he’ll get back up. soon. i’m sorry, i did all i could, let me sleep now, please… let… me… slee

Oh, no!

Despite the over-explaining in the note, the end of it is a fitting conclusion to the adventure. It’s cryptic, but not too much, and leaves the reader wanting more.

Speculation

I’m gonna try here to take educated guesses as to what happened in SCP-093. I know it’s not really analysis, but I’ll use it to prove a point later.

So, first of all, what is Him? All we really know about Him is from the last log, so let’s put all that together:

He’s obviously not human, as the description of his origin (“it came from between the folds of time and space and worlds and light and dark something that is but should not be”) kinda makes me think of “Lovecraftian Monstrosity.”
He is able to impersonate, or is, a god.
He created several extremely high-end technologies, including SCP-093.

Now that we know what He is, what happened to him? My two theories are:

He killed and impersonated the service technician to get into our reality, and when he was killed He was killed (boring ending).
He is still alive, and is somehow attached to SCP-093 (see below).

I find the second one more interesting, and I think it actually makes a couple of writing quirks that I’ve commented on so far seem justified.

Let me use an example. For the purposes of the example, let’s give names to the worlds involved in the story:

World 1 - The world accessed in the Red Test, and the “Hub World” for SCP-093.
World 2 - The world accessed in the Blue-Yellow Tests, and has fallen to Him and the Unclean.
World 3 - The world where Agent ██████████ from the recovered documents is from.
World 4 - The world where our Foundation is in.

Let’s establish a timeline. In World 1, He appears, possibly just a natural anomaly, like how SCP-173 is an anomaly native to our world. He takes over World 1, builds the 093 hub, and sets out to send 093’s to the other dimensions to bring Him to the other worlds.

World 2 is a world that got successfully invaded by Him and taken over.

World 3 is a world that got ahold of SCP-093 and apparently went on a thorough investigation of either World 2 or another Unclean world. However, the agent eventually realized what SCP-093 really did, and then broke it to He couldn’t get to World 3.

In World 4, AKA our world, we see that SCP-093 is really just a vehicle for Him. We now know that its usage could lead to the end of the world for all of us. So the Foundation does what it does best: it locks it up in a box, to make sure it doesn’t see the light of day.

This explains the narrative quirk of: “why didn’t the Foundation do more explorations of SCP-093?” Why didn’t they pass around 093 some more to see if they could get it to glow orange, or something? Why didn’t they send an MTF back into the Blue Test so that they could break down some of those stuck doors? Why didn’t they even test out the Unclean-killing revolver? The answer is simple: to open up SCP-093 again would be to risk the same thing happening to us.

This also explains the missing sadness property from the original documentation: it was there originally, but maybe He fixed it so people would be more likely to pick it up. This also explains why SCP-093 spins towards mirrors: it’s Him, trying to show us the way to let him into our world.

This also explains the very first line of the SCP:

See testing document SCP-093-T1 for outline of testing conditions.

They’re saying “if we’re testing this thing again, whatever you do, don’t open up a mirror portal.” As the first thing we see in the documentation, this is driving home the fact that the Foundation wants SCP-093 to stay buried.

I love these SCPs that you can speculate on, and SCP-093 does a fantastic job of facilitating this speculation. It’s close-ended enough to be a satisfying story, while open-ended enough to the reader to have a sense of wonder.

In short, SCP-093 is a timeless masterpiece. It not only presents a compelling story and a well-built world at a time when these things were rare, but also develops it all through subtle, masterful writing, and introduces some ambitious concepts that make the SCP truly shine. Really, this could’ve been a lot worse; NekoChris could’ve just said “let’s just make it roll towards mirrors now” during his rewrite and left it at that. Who knows where this skip would be if that happened? But instead, NekoChris made something dazzling. NekoChris made a classic.

Parting Shots

Of course, this isn’t perfect. Some problems I noted:

There’s some Series 1 bits that I mentioned above. These slow the skip’s momentum and could be executed better.
The tone of the exploration logs is all over the place, and definitely could use a run-through for clinicality.
The last document comes across as an info-dump of sorts. I think that it could be spread more across the logs.

However, what I came here to do is to figure out what SCP-093 does right, and how we, as writers, can learn from it. Here’s some key takeaways:

Don’t Show Your Entire Monster - This is a weakness of the SCP Format: it often forces you to show the entirety of your monster up-front. But when you can, gradually reveal your monster to keep the reader guessing as to what it is.
Create Threading Mysteries - Stuff like the mystery goo and the Unclean that recur are what trigger reactions out of the audience and keep them hooked.
Imagery - With clinical tone, it’s hard to pull off imagery. But when you can, the pictures and worlds you can create are fabulous.
Don’t Overwhelm the Reader - Using a “perspective character” can help to keep information flowing at an appropriate, but exciting pace.
Leave the story Open-Ended - I see too many skips today that don’t leave anything up to speculation. This can be a great strength and helps to keep up the mystery.

If you want to see a skip that takes these lessons from SCP-093 and follows them to a T, look no further than djkaktus’ excellent SCP-2935. It barely even lets the reader get a picture of the anomaly at first, it has a ton of recurring mysteries, the world is very fleshed out through imagery, the information is well-paced, and the ending is solid enough to be impactful but open enough to leave room for speculation.

There are some other similarities too, like the log structure, the apocalyptic theme and how the ending is put together. I don’t know if this was an intentional homage to SCP-093, or if it was coincidental (the number 2935 points me towards the former conclusion), but it’s a good example of something that follows SCP-093’s example.

In conclusion, when we write today, instead of looking at older skips through our new-author-eyes, I think we should instead look at our newer skips through the eyes of the old authors. We can learn a lot from the tried-and-true older skips.

The rabbit hole of unsafe Rust bugs

2023-12-16T00:00:00+00:00

unsafe code is not only tricky to get right. It’s also tricky to track down when things go wrong.

Here’s a question to ponder: do you think you are experienced enough at Rust to use unsafe code properly? If you asked me 24 hours ago, I would confidently say “yes, I am.” I would describe my Rust experience level as at least intermediate. On top of that, I have a little bit of an ego.

In fact, I fully support using unsafe in libraries! Many times, there’s no other way to do something!

You’re communicating with a system API or C library. Although, for the former, I would recommend seeing if using rustix or another system interface is an option.
You’re doing something that the borrow checker or type system can’t fully check your work on, even with workarounds like RefCell or Any.
There’s just some performance boost that’s simply not possible without an unsafe workaround. I would strongly recommend benchmarking this claim before implementing it in a publicly used crate.

While I’m not against unsafe, I am for really checking and validating that unsafe. You need to be making sure it’s doing what it needs to be doing, and that it’s doing that thing right. The hazard with unsafe isn’t just that it can break your crate. It’s that it can break other crates, in a way that’s very, very difficult to trace back to your original crate.

You know how I said I had a chip on my shoulder? In the past week, I’ve had two soundness errors reported. For the same crate, no less!

One of those soundness errors is a fairly routine linked list rigamarole with an associated discussions about public API. The other one is a frantic chase through several different crates, where things are never as they seem. This blog post focuses on the second one, as I find it more entertaining and maybe we can even shoehorn a moral in there somewhere.

So let’s ask ourselves: what do we open ourselves up to when we add unchecked unsafe to our code.

I mention crates that are created by and maintained by other people in this post. They have done nothing wrong: all of the bugs were introduced by me, notgull. It should go without saying, but please do not seek out or harass these other people.

Event Listener Escapade

Let’s introduce our cast of characters.

event-listener is a fairly primitive crate in the smol ecosystem. It’s main responsibility is communications between tasks; it essentially handles the “if one thing happens in a task, wake up another task” use case. It’s used to implement locks and channels, among other things.

event-listener also has a lot of unsafe code. In my opinion, this is a good thing: it means that other crates that depend on it don’t need to have that unsafe code for themselves. I’ll also be the first to admit that a lot of the unsafe code is unnecessary. Some of it exists because we’ve benchmarked it to be faster, and some of it exists because there isn’t really a better way to implement, say, a limited-contention spinlock in Safe Rust.

A few days ago, one of our maintainers reported a bug. Most of the time, soundness bugs have a specific flavor, like bitter licorice. This bug didn’t have that flavor, at first glance. It was like biting into the sweet apple of a simple off-by-one error, with an aftertaste of smooth, smooth WebAssembly. One fateful day, GitHub Actions started failing with a strange error.

$ wasm-pack test --node --no-default-features --features portable-atomic
Running tests/notify.rs (target/wasm32-unknown-unknown/debug/deps/notify-0ef5209d8abd9615.wasm)
Set timeout to 20 seconds...
Executing bindgen...                              
                                                  
running 9 tests

panicked at /home/runner/work/event-listener/event-listener/src/no_std.rs:639:48:
index out of bounds: the len is 0 but the index is 4

Here is the full log, in case you want to try to take a crack at this bug for yourself.

Debugging Demonology

At first glance, I ran this code on my laptop and wasn’t able to replicate the issue.

$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests pass>

I took one glance at the WASM bits and the portable-atomic feature and decided that it must be some kind of strange one-in-one-thousand race condition. It was a Thursday and I had work to do, so I decided to revisit it on the weekend. The weekend came and, after re-running the Actions workflow, the bugs were still there.

Weird, I thought to myself. If it was a race condition, it shouldn’t be happening consistently. So I needed to take a closer look at what was happening here.

While reading through the logs, I realized that the list that the out-of-bounds error was happening on was initialized like this:

/// Create a new, empty list.
pub(crate) fn new() -> Self {
    Self {
        listeners: alloc::vec![Entry::Sentinel],
        <snip: irrelevant fields>
    }
}

This list never has elements removed from it. It works a lot like the slab crate does; it leaves those slots open for future entries to fill. Remember that “out of bounds” error?

index out of bounds: the len is 0 but the index is 4

“len is 0”? The length of this list can only ever be one or more. So unless I’m accidentally calling remove() or truncate() on that list at some point, the length should never be zero.

On top of that, I noticed this little detail in the logs as well, for another failed test:

called `Result::unwrap()` on an `Err` value: Full(Notify { count: 1, is_additional: true })

This operation results from a concurrent queue push operation, where the queue is initialized like this:

pub(super) fn new() -> List<T> {
    List {
        queue: concurrent_queue::ConcurrentQueue::unbounded(),
        <snip: irrelevant fields>
    }
}

Unbounded queues can never be full. It doesn’t happen; they just allocate more memory to make room. At this point I curled my nose as the bitter aroma of memory corruption started to waft my way.

“Maybe it’s a rustc bug” I say to myself, still in the “denial” phase. I had to keep digging deeper; hopefully I can find the actual cause of the error and set up a minimal reproduction of it.

Damaged Dependencies

At this point, it occurred to me that one of the differences between my machine and the Actions virtual machine is that I have a Cargo.lock with some older dependencies in it. In most of my projects I don’t commit Cargo.lock to Git, even though it’s the default now.

So I back up my current Cargo.lock as a “last good state” (foreshadowing for later) and try the tests again.

$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests succeed>
$ cargo update
    Updating crates.io index
    Updating concurrent-queue v2.3.0 -> v2.4.0
    Updating crossbeam-utils v0.8.16 -> v0.8.17
    Updating futures-lite v2.0.1 -> v2.1.0
    Updating itoa v1.0.9 -> v1.0.10
    Updating js-sys v0.3.65 -> v0.3.66
    Updating libc v0.2.150 -> v0.2.151
    Updating once_cell v1.18.0 -> v1.19.0
    Updating portable-atomic v1.5.1 -> v1.6.0
    Updating proc-macro2 v1.0.69 -> v1.0.70
    Updating ryu v1.0.15 -> v1.0.16
    Updating serde v1.0.192 -> v1.0.193
    Updating serde_derive v1.0.192 -> v1.0.193
    Updating syn v2.0.39 -> v2.0.41
    Updating wasm-bindgen v0.2.88 -> v0.2.89
    Updating wasm-bindgen-backend v0.2.88 -> v0.2.89
    Updating wasm-bindgen-futures v0.4.38 -> v0.4.39
    Updating wasm-bindgen-macro v0.2.88 -> v0.2.89
    Updating wasm-bindgen-macro-support v0.2.88 -> v0.2.89
    Updating wasm-bindgen-shared v0.2.88 -> v0.2.89
    Updating wasm-bindgen-test v0.3.38 -> v0.3.39
    Updating wasm-bindgen-test-macro v0.3.38 -> v0.3.39
    Updating web-sys v0.3.65 -> v0.3.66
$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests fail again>

“Aha! It’s one of the dependencies!” I shout aloud to myself, fully aware no one is listening. “It’s not my fault!”

Put yourself in my shoes for a second. It’s probably portable-atomic, right? I mean, the error only happens when portable-atomic support is enabled. It’s a crate that adds a polyfill for the core::sync::atomic module that works on embedded platforms without atomics. It’s very thorough and has support for many different architectures and platforms. Since it has a lot of unsafe, that makes it very easy to blame.

So, let’s update it and see what happens.

$ cp Cargo.lock.old Cargo.lock
$ cargo update portable-atomic
    Updating crates.io index
    Updating portable-atomic v1.5.1 to v1.6.0
$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests succeed>

The tests passed… I’m sorry, what?

So it’s a dependency update… but not in portable-atomic? Weird, I guess I can run through the other dependencies we updated and see which one is the squeaky wheel. Let’s start with concurrent-queue, it seems innocent enough.

$ cp Cargo.lock.old Cargo.lock
$ cargo update concurrent-queue
    Updating crates.io index
    Updating concurrent-queue v2.3.0 -> v2.4.0
$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests fail>

At this point, I did a double take at my monitor. What?

concurrent-queue is a smol crate that provides a queue that can be accessed concurrently. It’s significantly more efficient than having a Mutex around a non-synchronous queue, especially when multiple threads are pushing to or reading from the queue. It also has a lot of unsafe code which, under normal circumstances, I would be happy to blame.

But, I was the one to push that update from v2.3.0 to v2.4.0. The only real change was getting rid of a heap allocation. This didn’t add or remove any unsafe code; it just moved some fields onto the stack instead of the heap… actually, it was still in the heap, since in event-listener the queue was wrapped with an Arc, which allocates on the heap.

Still, if you’re looking for someone to blame, concurrent-queue seemed like it was caught red-handed. The build broke when I updated it, that was the smoking gun!

Goose Chase Gallery

My attention was piqued when I noticed that I’d forgotten to add WebAssembly tests for concurrent-queue. If this was a concurrent-queue issue, it would be hidden by these lack of tests. My suspicions were confirmed when I ran tests using wasm-pack and saw that, yes, they were failing with portable-atomic being enabled.

I began to settle in for the grueling task of reviewing atomics to make sure that all of the operations were correct. Especially for a crate as intricate as concurrent-queue, this could take weeks. Except… let me check those tests again.

Oh wait, the only tests that are failing are the ones that try to spawn threads. Of course they fail, you can’t spawn threads on WASM! Let’s mask those tests out, and..

Wait, the tests work? Hey, how’s that possible?

Maybe… maybe the leftover tests aren’t good enough. Yes, maybe the bug I’m looking for isn’t covered by these tests, so I need to write more tests. So I need to finally figure out how web workers work. Then I’ll need to rewrite my tests, nay, my test harness, to properly run all of these test cases?

Alternatively, maybe we have the wrong guy.

`miri` Magic

Have you ever watched a police procedural? Usually, they play a trick on you in the middle of the episode. A crime happens, and after some detective work they find a guy who looks like they did it. They have evidence! The smoking gun! But our protagonist, a rookie cop who’s flexible about his position on torturing civilians, doesn’t think they did it. But their completely reasonable qualms that might come up during trial are ignored by the rest of the police force. Then the protagonist confronts the actual perpetrator, who’s usually some poorly developed background character, and brings him to justice.

In this metaphor I’m the police force, and I thought I’d caught concurrent-queue red-handed. But something didn’t sit right.

Before I got too deep into concurrent-queue’s innards, I realized something. In my CI, I wasn’t running miri tests for the case where portable-atomic was enabled. This meant that, if this was a memory corruption error, it wasn’t being caught by miri.

This introduced a thought into my head for the first time, a thought I should’ve had three days ago. Maybe it isn’t strictly a WASM issue?

So I ran miri with portable-atomic support, and sure enough:

$ cargo miri test --no-default-features --features portable-atomic --tests

<snip: miri noise>

test drop_non_notified ... error: Undefined Behavior: constructing invalid value: encountered an unaligned reference (required 128 byte alignment but found 16)
   --> /home/jtnunley/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/lib/rustlib/src/rust/library/core/src/ptr/mut_ptr.rs:367:57
    |
367 |         if self.is_null() { None } else { unsafe { Some(&*self) } }
    |                                                         ^^^^^^ constructing invalid value: encountered an unaligned reference (required 128 byte alignment but found 16)
    |
    = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
    = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
    = note: BACKTRACE:
    = note: inside `std::ptr::mut_ptr::<impl *mut event_listener::Inner<()>>::as_ref::<'_>` at /home/jtnunley/.rustup/toolchains/nightly-x86_64-unknown-linux-musl/lib/rustlib/src/rust/library/core/src/ptr/mut_ptr.rs:367:57: 367:63
note: inside `event_listener::Event::try_inner`
   --> /home/jtnunley/Projects/smol-rs/event-listener/src/lib.rs:443:18
    |
443 |         unsafe { inner.as_ref() }
    |                  ^^^^^^^^^^^^^^

Specifically, I noticed this error:

required 128 byte alignment but found 16

What this is telling me is that we are trying to dereference a pointer that isn’t properly aligned. So, where is this pointer coming from?

/// Return a reference to the inner state if it has been initialized.
#[inline]
fn try_inner(&self) -> Option<&Inner<T>> {
    let inner = self.inner.load(Ordering::Acquire);
    unsafe { inner.as_ref() }
}

Okay, this is just the pointer we’re storing in the self.inner atomic variable. How are we calculating that pointer?

// Allocate the state on the heap.
let new = Arc::new(Inner::<T>::new());

// Convert the state to a raw pointer.
let new = Arc::into_raw(new) as *mut Inner<T>;

// Replace the null pointer with the new state pointer.
inner = self
    .inner
    .compare_exchange(inner, new, Ordering::AcqRel, Ordering::Acquire)
    .unwrap_or_else(|x| x);

So, inner is just the return value of an Arc allocation? Weird. The standard library’s types are usually pretty well-tested. Not to mention, I’m sure that other programs make use of dereferencing the into_raw too, and they would have seen this error before me right now. Let me check my imports…

Oh no.

#[cfg(feature = "portable-atomic")]
use portable_atomic_util::Arc;

Oh no.

Arc Apocalypse

portable-atomic-util is a sort of “side crate” to portable-atomic. It’s where you put things that are too specific to be put in portable-atomic, but are probably still useful.

One of the tools in portable-atomic-util is an implementation of Arc. It’s similar to the one in the standard library, but instead of using atomics from the standard library it uses portable-atomic. Unfortunately it’s not a 1:1 copy of the one from the standard library, as the standard library makes use of unstable features that can’t be used from normal library crates.

So, let’s take a look at its into_raw implementation. It looks a little like this:

// The inner heap allocation of an `Arc`.
#[repr(C)]
struct Shared<T: ?Sized> {
    /// The reference count of the `Arc`.
    header: Header,

    /// The value that is being reference counted.
    value: T,
}

impl<T> Arc<T> {
    #[must_use]
    pub fn as_ptr(&self) -> *const T {
        // Get the raw pointer.
        let ptr = self.shared.as_ptr() as *mut u8;

        // Add the size of the header so that it points to the value.
        let new_ptr = strict::map_addr(ptr, |addr| addr + mem::size_of::<Header>());

        // Cast the pointer to the correct type.
        strict::with_metadata_of(new_ptr, self.shared.as_ptr() as *mut T)
    }
}

It’s pretty simple. Our shared pointer points to an instance of the Shared structure on the heap, and we want it to point to the value field. That way, when the user dereferences the pointer, they get the underlying T.

Unfortunately we can’t use addr_of_mut like the standard library yet. It’s stable, but portable-atomic-util’s MSRV is lower than the version where it was introduced. So we do the math manually. The value will be after the Header, so just add the size of the Header to get the pointer. Right?

Wrong.

What the author of this code failed to consider is alignment. A lot of structures need to have their address be a multiple of a certain number, usually the size of their largest field. The reasons for this are related to the underlying CPU architectures and I can’t really get into that. This article is looking pretty long as it is.

However, the ConcurrentQueue type has an alignment of 128. This is related to how cache lines are implemented on the x86 architecture. When you have frequently changed atomic data, you want to put in a separate cache line than other data. That way, the processor won’t need to reload that other data too frequently. No, we really don’t have time.

So, it is possible for Shared above to be laid out with the Header at the start, then have some number of padding bytes before the actual value field. This means that the pointer addition from above is now pointing into padding, rather than the actual T value. That explains the memory corruption.

It also explains why it’s only started to cause undefined behavior now. In v2.3.0 of ConcurrentQueue, the data with an alignment of 128 was stored inside of a heap allocation. Since heap allocations have an alignment of the platform word size (in the case of 64-bit architectures, 8), there was no padding between the Header and the value here. It was only when that heap allocation was removed and the alignment was increased that this problem appeared.

Say, who wrote this code? I’ll need to find them and give them a piece of my mind. git blame can tell us who, exactly, is responsible for this mess.

Egads! It was me all along!

This bug is no longer in portable-atomic-util. I filed a PR which has now been merged, and the versions of portable-atomic-util with that bug have been yanked from crates.io.

To come full-circle, let’s see if our tests work with the newest version of portable-atomic-util.

$ wasm-pack test --node --no-default-features --features portable-atomic
<snip: tests succeed>

Awesome!

Parting Shots

So, what’s my conclusion? What did we learn from this little escapade?

I’d like to bring your attention to the source of the original soundness bug.

#[must_use]
pub fn as_ptr(&self) -> *const T {
    // Get the raw pointer.
    let ptr = self.shared.as_ptr() as *mut u8;

    // Add the size of the header so that it points to the value.
    let new_ptr = strict::map_addr(ptr, |addr| addr + mem::size_of::<Header>());

    // Cast the pointer to the correct type.
    strict::with_metadata_of(new_ptr, self.shared.as_ptr() as *mut T)
}

Eagle-eyed readers might have noticed that this code has no unsafe. Not even a line. Sure, it does some risky pointer math, but that’s considered safe in Rust.

Also, keep in mind that concurrent-queue had no bugs introduced either. It just changed the alignment of a structure from 8 to 128. That can be done in safe Rust; hell, it isn’t even a minor change according to SemVer.

Here is the actual unsafe code that caused this unsoundness.

#[inline]
fn try_inner(&self) -> Option<&Inner<T>> {
    let inner = self.inner.load(Ordering::Acquire);
    unsafe { inner.as_ref() }
}

It’s not unsound; it’s not even incorrect. This code was checked, audited and did everything that it was supposed to do. It was a combination of two pieces of 100% safe code from another crate that caused this bug to happen.

When I begun this article, I talked about how you need to check your unsafe code. What I wanted to prove is that you can’t just check your unsafe code. You need to check each and every line of safe code too. Safety is non-local, and a bug in safe code can easily cause unsound behavior in your unsafe code if you’re not careful.

For me, I’m going to be extra careful when I write new unsafe logic. I advise that you do the same.

Evaluating new software forges

2023-12-15T00:00:00+00:00

What options are there other than GitHub?

Oh boy, I sure do love contributing to open source software on the largest software forge in the world! I hope they haven’t started down the slow and painful process of enshittification by following vague, ill-defined industry trends!

Wait, what’s that? Computer! Enhance!

Well, I guess I’m ready to find a new forge!

Software Host Hellscape

In all seriousness, I’ve been looking to move off of GitHub for a while now. Let me be clear, GitHub is still far and away the best website for open source discovery. Not to mention, its CI offerings are very nice, especially for something free. Yes, there are better paid CI offerings, but for something that costs zero dollars I’ve found it incredibly useful.

However, one thing has made me skeptical of GitHub is its “Copilot” offering. I’ll admit, I was in the beta program for Copilot, and found it really neat. Being able to write large amounts of code from small comments was very nice, even if it was really bad practice.

Then I found out it was training on GPL-licensed data, which left a pretty bad taste in my mouth. In addition to the fact that I’m increasingly uncomfortable with hosting my free software on a closed source forge, run by Microsoft.

Let’s take a look at everybody else.

GitLab Gauss

GitLab is the original GitHub competitor. The Linux to Microsoft’s Windows, or the MariaDB to Oracle’s MySQL. This has made it the most popular GitHub competitor by far, by virtue of people vocally quitting GitHub in favor of [GitLab].

Unfortunately, I didn’t really consider GitLab when I was finding a place to move. First of all, they aren’t actually open source. They’re “open core”, which, I admit, is better than closed source. However, like I said, I’m uncomfortable building free software on infrastructure that isn’t.

I know I can download GitLab and set it up on my own server. However, I’m a software developer, not a sysadmin. I want to spend my time developing software, not putting out fires and paying AWS bills for the rest of time.

Also, GitLab has adopted the unfortunate strategy of “following along with whatever GitHub does”. They’ve tried to jump onto the bandwagon so frequently, they’ve gotten splinters. For instance, what happens when we check their homepage?

Computer! Do the thing again!

Good golly, it’s even the same wording! Yeah, I’ll pass.

SourceHut Scramble

sr.ht takes the opposite approach as GitLab. Instead of trying to follow along with GitHub’s trends, it’s elected to do go in the other direction. Whenever GitHub does something, SourceHut does the exact opposite.

Pull requests? Too centralized, let’s construct a suitable code contribution system around email. Discussion? Why not IRC, it’s been around since the Bronze Age. Get rid of Mercurial support? Not interested.

I really like SourceHut. When you go to their homepage, they’re not showing off their fancy CSS effects or telling you about their AI offerings. They give you a simple user interface and some of the projects they host.

There was also much to impress me. Their CI offerings are better than GitHub, which alone justified me paying the humble $2/month price tag. Rather than needing a complicated YAML file to run a CI system, it’s just cloning Git repos and running commands. It’s delightfully simple yet powerful. Having native BSD and Plan9 runners doesn’t quite make up for its inability to run Windows, but I’m sure I can work around that.

Not to mention, SourceHut has the second best repository discovery system. When I go to sr.ht’s “explore” tab, I’m immediately greeted by a slew of interesting projects. Whether it’s a powerful Forth dialect that brings a lot of genuinely exciting ideas to the table, or a tiny C11 compiler written in simple ANSI C, I’m always amazed whenever I open up that tab.

I liked it so much that I announced that I was moving my personal projects to SourceHut. However, after moving my theo project to SourceHut, I found myself dissatisfied with a few things.

For one, the email-based workflow was a lot clunkier than I expected. In theory, building code contribution on top of a standard protocol that’s been around since the 60’s sounds like a good idea. In practice, it’s a lot clunkier than you’d expect, especially since most modern email clients are simply not built to read and write code.

After trying it for myself I can see that it might turn off a lot of people from contributing. I’m already losing a lot of potential contributors by moving off of GitHub. I don’t need those remaining contributors to also be turned off by a workflow completely different than what they’re probably used to.

Still, I can imagine this workflow working for many people, especially ones who already have a decent setup for email-based projects like Linux.

Codeberg

Codeberg is a public instance of Forgejo, which is in turn a fork of Gitea. It’s got a pretty nice interface similar enough to GitHub’s. It’s got the familiar pull-request-based contribution interface. The CI is good enough, I suppose. Docker containers aren’t the best CI environment, but I can certainly think of worse.

So what’s not to like?

My main problem is that Codeberg has a very limited CI capacity, and I have a lot of projects that require significant testing. theo, for instance, requires these things to be tested:

Make sure it compiles on Windows, Mac, Linux, WASM, Redox, and whatever oblique platform people run Rust on nowadays.
Check the various different backends that theo supports: pure software rendering, wgpu and OpenGL. Not to mention all of the different interfaces to OpenGL, so wgl, GLX, EGL…
Check formatting and linting.

…which doesn’t even cover testing. For rendering frameworks like theo, you want to have some pre-defined rendering programs that render your code to images. That way, you can regenerate these images and compare against an existing set of images in order to check for regressions.

This isn’t a practical concern, although it really should be. It’s a moral concern. You have an organization like Codeberg, donating a significant amount of time and resources to try to make a positive difference in the world of software. Now, here I am, sucking up all of those compute resources for my insignificant little projects.

Of course, while pondering this moral concern, I realized that I’ve locked myself into a Catch-22. I can’t use any independent project’s CI because of my concerns that I would drain too many of their resources. On the other hand, I can’t use any large company’s CI because I don’t want to host my project with a large company. I can’t self host, because that would be a pain.

Would it?

Self-Hosting Gitea

I said I didn’t want to self-host. I worked in IT for two years, so I’ve already gotten my fill of fighting with both servers and people.

However, in a Discord I’m in, an acquaintance of mine talked about how they set up Gitea and Drone CI on a school Kubernetes cluster they had access to. I mentioned my predicaments in finding a forge service, and they said that it was only two configuration files.

That tempted me. Not enough to deal with the absolute nightmare that is Kubernetes, but enough for me to rent out a couple of Lightsail servers to experiment.

I’d like to say that I set up the entire thing in a weekend, but it wasn’t that simple. Sure, it was easy enough to install Gitea and Drone in Docker containers. Sure, it wasn’t too hard after that to set up my DNS records to forward src.2023.notgull.net to that new serer. Yeah, it’s probably hacky to set up my CI system on a public cloud, but as long as I keep people from abusing it that aren’t me it shouldn’t be too bad, right?

Of course, I forget my crucial weakness: my perfectionism. Sure, my site looks pretty good… but it looks a bit ugly. Let’s play around with themes for three days until I find one I like. Oh, the logo doesn’t look good with my new theme. Let’s convert the MS paint drawing I call my avatar into SVG and then set it up to be this site’s logo. Oh, Gitea has a weird templating system; let’s compile it from scratch!

At the end of this week(!)long process, I had a somewhat functional Git forge, with the following features:

A decent Linux Drone CI setup.
A Dependabot clone for automatic updates.
A system that allows everyone to create an account and open issues, but doesn’t allow people to create repos or mess with the CI.
- If you want to create a repo for PR’s sake, email me at dev at notgull dot net and I can set you up.
A mirror to GitHub, so I can still take advantage of their code discovery.
- Yeah yeah, I know, but keeping an arm’s length from GitHub is a win in my book.

I’ve uploaded a lot of my code to there, and it seems to be working well so far. I have to admit, it is somewhat liberating to have full control of how your code is forged.

Parting Shots

I still consider this to be in the “experimental” stage. If it ends up being too inconvenient or too expensive, I’ll probably move it somewhere else. Still, having my own space for code that I can do whatever I want with is very nice. Let’s hope it keeps working well and this blog post doesn’t age like milk.

All photos are from public websites and fall under free use, as this is a review.

My new, remote-access Rust development setup

2023-11-12T00:00:00+00:00

I’ve set up a new system for Rust development work.

I work on quite a few crates in the Rust ecosystem. Since I started taking Rust seriously back in 2021, I’ve been using the same laptop that I’ve used since high school.

It’s actually a pretty beefy laptop, all things considered. It has an i5 CPU and 16 GB of RAM. It ran Ubuntu for the majority of its life but now runs Alpine Linux. These stats were definitely overkill for a high schooler playing around with Python. Even now, it still works with smaller codebases. Honestly, it’s an exceptionally first world problem to even be complaining about.

However, I’ve started to hit the limits of this computer. 16 gigabytes of RAM is a lot when you’re dealing with codebases like smol. Even if you put all of smol’s subcrates together, rust-analyzer barely uses up four gigabytes.

However, lately I’ve been working on codebases that overflow my limits. winit is a medium-sized codebase at around 25KB lines, but with all of the dependencies it makes rust-analyzer slow to a crawl. x11rb has a lot of automatically generated code; I’ve actually experienced a kernel panic because rust-analyzer took up all of my memory processing x11rb, including an additional 16 gigabytes of swap space.

Not to mention, once my laptop touches swap space, it might as well be over. Memory access slows to a crawl, and the latency between my thoughts and my applications might as well be dial-up.

Time for an Upgrade

Since I started this blog in March of this year, I’ve managed to trick a company into giving me money in exchange for writing code for them. Around Prime Day, I decided to burn a paycheck on an upgrade.

My goal was to build a server that I can access remotely. I move around a lot; I like to write code at various hackerspaces and coffee shops around the city. It’s a nice, social environment where I can talk to (or at least be around) people while I code. I’m already an introvert, so I don’t want to be cooped up inside if I want access to serious computing power.

The remaining goal was absolute overkill. I don’t want to have to upgrade this machine for the forseeable future. Therefore, I’m going all-out on hardware and software.

The Hardware Specs Section

I built this computer around the AMD Ryzen 9 5950x processor. It was half-off for Prime Day, which is what prompted this entire process. It has 32 cores and is reasonably fast. I’m aware I could go higher, but I don’t want to break my bank account for something I’m not being paid for.

Don’t get me wrong; my laptop’s CPU was already good enough for my use case. With Rust’s incremental builds, I very rarely spend more than a few seconds waiting for feedback from rust-analyzer. Still, from-scratch compiles took a few minutes, especially for larger crates.

With 32 cores, I can easily parallelize crate builds. cargo install ripgrep completes in less than a minute. Combined with an absolutely overkill 128 gigabytes of RAM, I can build any crate I want to in record time.

My workload isn’t GPU intensive, so I grabbed an AMD Radeon 570x to use as a GPU to tie the build together. I combined this with an ASRock motherboard, a CoolerMaster chassis and liquid-cooling system, and a 4TB SSD to tie the rest of the computer together. The end result is a system that I don’t intend to use directly, but will be very nice to use remotely.

Putting It All Together

I’d like to muse briefly on putting this computer together. I’ve never actually built a computer before. Usually, when I’m in need of a new computer, I don’t have time to go through the process of figuring out which parts go where. So I’ve only used prebuilt systems until now.

I’ve heard from some people that putting together a PC is like building Legos. While this is true in some respects, keep in mind that these are 2000 dollar legos, where in some cases it might be unclear what part goes where. This leads to a lot of stress; especially around installing the CPU. One bent pin means that you have to throw out the CPU.

Thankfully, on attempt number three, my friend and I successfully managed to put the CPU in the socket and install the liquid-cooling system. Everything else screwed into place relatively easily. The motherboard goes into the chassis, and everything hooks into the motherboard from there.

However, I made two crucial mistakes. First, I ordered a 3U chassis where I actually needed a 4U chassis. I also needed a 750W power supply instead of the 500 W one I originally calculated I needed. Amazon has a decent return policy, so I was able to return those parts and get the ones I actually need.

Finally, after putting it all together, it refused to boot! We thought we’d installed the CPU wrong. After crawling over the manuals and forums for the motherboard a few times, we figured out that e needed to plug in a keybard. Weird. Anyways, one Alpine Linux install later, I had a working PC.

Utilizing the System

I’ve set up this PC in the attic. Thankfully, the liquid cooling is quiet enough so that I don’t hear it when I’m trying to sleep.

Now, I want to be able to access this from anywhere. Like I said, I want to be able to use this computer like I’m sitting down at it, even if I might be across the country. Originally, I was going to set up an SSH tunnel between my laptop and this server, using an AWS Lightsail instance as an intermediary.

However, then I found out about Tailscale (not sponsored), which fit my use case much better.

So, I’ve scaled my tail. I’ve hooked up my new server, my phone and my laptop to a tailnet. SSH is a breeze and setting up other services is a cinch as well.

I’ve been getting used to code-server. In fact, I’m writing this in there right now! I’ve tried to get used to Neovim and Emacs and other editors like that. However, I’m just too used to the VS Code workflow at this point, especially since using it for work. Generally, I don’t notice a difference between code-server’s interface and VS-Code.

Desktop Dismay

However, there is one cinch. When testing out winit, I need direct GUI access. Remote X11 just won’t cut it, especially for rendering.

I’ve installed a VNC server on my server, and I can just tab over to my VNC client whenever I need to run a GUI application. Still, it’s a somewhat awkward workflow. I’m open to new ideas.

Conclusions

There isn’t a moral to this story, I just wanted to talk about my new remote workflow. I hope this overview inspires similar remote flows in the future.

Recreating concurrent futures combinators in smol

2023-10-22T00:00:00+00:00

futures comes with many additional combinators that smol doesn’t have out of the box. We can rebuild them, better.

Whew, it’s almost been a month since my last blogpost here. This was because I was spending time doing research and testing, and not because I lost the PGP key that allows me to upload to this site. No sir, how could anyone be that irresponsible?

…or maybe I was just using the PGP key as an excuse not to write? It’s not like I’m being paid to psychoanalyze myself in front of you people.

It doesn’t matter, we’re back! Let’s talk about smol.

The Problem with `futures`

futures was originally released in 2016 to provide an implementation of asynchronous programming for Rust. In the time since, it’s accumulated a lot of baggage. Many of its combinators have been superseded by the async/await syntax, meaning that a large amount of its API surface isn’t relevant anymore.

For instance, take the Map combinator. It takes the value of some Future and maps the return value through some closure.

use futures::prelude::*;

let fut = async { 1 };
let mapped_fut = fut.map(|x| x * 2);

In the pre-2018-edition days, these combinators were the only way to manipulate the value of a Future. They were completely necessary for using these asynchronous values back in the day. Nowadays, we can just wrap the original futures using async/await and treat it more like a normal expression.

let fut = async { 1 };
let mapped_fut = async { fut.await * 2 };

Therefore, in this brave new post-async world, many of these combinators became unnecessary. futures-lite, one of the core components of smol, was created to address this new reality.

futures-lite explicitly ignores all combinators that can be implemented using async/await or features that have already been moved into the standard library. This leaves behind a small, clean subset of the API that builds fast.

A semi-frequently asked question I see goes along the lines of: “I was porting my application over to smol, but I noticed that it doesn’t have for_each_concurrent or buffered. Is this API excluded purposely?”

This is a reasonable question. The short answer is “yes”, and the medium answer is “these concurrent functions have small but frustrating problems that futures-lite avoids by not implementing them”. This article is the longer answer.

Concurrency Conundrum

I would argue that the concurrent Stream stream combinators mentioned above are a code smell. Well-formed production-ready code should not use for_each_concurrent or buffered. If I knew how compilers worked, I would suggest a Clippy lint that would flag these functions as a warning.

The for_each_concurrent function is called like this:

let my_stream = /* ... */;
my_stream.for_each_concurrent(
    None, // Limiter parameter, not important for now.
    |x| async move { do_something(x).await } // Closure to run for each element.
).await;

To massively oversimplify, for_each_concurrent does this:

use futures::prelude::*;
use futures::stream::FuturesUnordered;

let my_stream = /* ... */;

// Create a list to store all of the futures.
let mut futures_list = FuturesUnordered::new();

// Get all of the values from our stream.
while let Some(x) = my_stream.next().await {
    // Push the future to the list.
    futures_list.push(async move { x + 5 });
}

// Wait for all of the futures to complete. FuturesUnordered polls each future
// in order and returns their results.
futures_list.for_each(|()| {}).await;

FuturesUnordered is sort of an unholy combination of an executor and a Stream. It collects a bunch of Futures into a list, then maintains a queue of which Futures are ready to be polled. Once a Future returns that it is Ready, it returns that Future’s value.

This means that, every time you call for_each_concurrent, it creates an entire new executor, runs the Stream to completion on it, then discards that executor entirely. buffered is implemented in a similar way.

It’s bad for a couple of reasons. Most async runtimes already provide an executor. tokio provides one out of the box, and smol encourages you to create and optimize your own. By using for_each_concurrent or buffered, you are essentially ignoring your previous executor in order to spawn a short-lived temporary executor.

In addition to the resources that are wasted on the new executor, it’s often less efficient than async runtime executors. tokio and smol support options to let you offload tasks on other threads or handle contention more efficiently. In contrast, FuturesUnordered is a relatively naive executor that is completely unaware of its surrounding runtime.

Not to mention, FuturesUnordered comes with a few footguns that make it impractical for common use cases.

Make your own, better combinator

In smol, you can emulate these use cases somewhat easily. First, you need to create an Executor and execute your features in its context.

let ex = smol::Executor::new();

ex.run(async {
    // The code written below will take place in this context.
}).await;

You can emulate for_each_concurrent by turning every future in the stream into a task, then awaiting all of those tasks. Here’s how it looks if you don’t have a task limit:

use smol::prelude::*;

let my_stream = smol::stream::iter(vec![1u32, 2, 3]);

// Spawn the set of futures on an executor.
let handles: Vec<smol::Task<()>> = my_stream
    .map(|item| {
        // Spawn the future on the executor.
        ex.spawn(do_something(item))
    }).collect().await;

// Wait for all of the handles to complete.
for handle in handles {
    handle.await;
}

Here, we spawn every future involved onto the executor. We then take all of the task handles and collect them. Since we are running inside of the executor, all of these tasks will be run in parallel. Finally, we just .await on each handle to wait for all of the tasks to complete.

The best part is that the allocation, the Vec<smol::Task<()>>, isn’t even necessary. It could be one-time allocation that is just extended to hold the tasks.

Generally, it doesn’t matter how many tasks are spawned onto the global executor. In contrast to the mini-executor that for_each_concurrent spawns, the global executor is designed to handle large numbers of tasks. However, if you still want to impose a resource limit, you can use a Semaphore.

use smol::prelude::*;
use std::sync::Arc;

let my_stream = smol::stream::iter(vec![1u32, 2, 3]);
let my_limit = 5;

// Semaphore for limiting the number of tasks.
let semaphore = Arc::new(smol::lock::Semaphore::new(my_limit));

// Spawn the set of futures on an executor.
let handles: Vec<smol::Task<()>> = my_stream
    // We use using `then` now, since we need to `.await` for the 
    // semaphore to have a permit available.
    .then(|item| {
        // Borrow the semaphore and executor.
        let (ex, semaphore) = (&ex, &semaphore);
        async move {
            // Wait for a semaphore permit.
            let permit = semaphore.acquire_arc().await;

            // Spawn the future on the executor.
            ex.spawn(async move {
                // Run our future.
                do_something(item).await;

                // Drop the permit to let another task run.
                drop(permit);
            })
        }
    })
    .collect()
    .await;

// Wait for the remaining handles to complete.
for handle in handles {
    handle.await;
}

This works by having each task borrow a Semaphore permit. The semaphore is sort of like a Mutex that can be locked by multiple parties at once, up to a certain limit. Once it runs out of permits, this code doesn’t spawn a task until one of the tasks completes. The permit is moved into the task and is dropped once the computation completes.

The then-stream above is also practically the equivalent of buffered. It yields tasks that can then be awaited to get their results.

// snip: semaphore and stream setup

// This time, we do something else that maps the value to another.
async fn do_something_else(x: u32) -> u32 { x + 1 }

// Get a `Stream` of tasks that can be `await`ed to get their value.
let buffered_stream = my_stream
    .then(|item| {
        // Borrow the semaphore and executor.
        let (ex, semaphore) = (&ex, &semaphore);
        async move {
            // Wait for a semaphore permit.
            let permit = semaphore.acquire_arc().await;

            // Spawn the future on the executor.
            ex.spawn(async move {
                // Run our future.
                // NEW: This now returns a value.
                let result = do_something_else(item).await;

                // Drop the permit to let another task run.
                drop(permit);

                // NEW: Return the result of the inner future.
                result
            })
        }
    });

// NEW: Because the stream uses an unpinned async closure,
// we have to pin it.
smol::pin!(buffered_stream);

// NEW: We can now wait on this stream for its values.
while let Some(task) = buffered_stream.next().await {
    println!("Value: {}", task.await);
}

This is all practically more efficient than buffered while giving you much greater control over how it runs.

Parting Shots

Unfortunately there’s not much documentation for the fact that for_each_concurrent and buffered spawn their own separate executors. Raising awareness of proper async code is powerful, in my opinion, as it unlocks a whole new world of computations for intermediately experienced Rustaceans. I hope this makes it clearer what should be happening in well-formed code.

Eyra is an interesting Rust project

2023-09-25T00:00:00+00:00

In the eternal quest to rewrite everything in Rust, even the C standard library isn’t safe from carcinisation.

Modern Rust programs are, for the most part, written mostly in Rust. For networking applications, the entire asynchronous stack is Rusty; no libuv in sight, only mio and polling. There is a robust rendering stack based on tiny-skia and cosmic-text. Even if you need FFI, the story is still pretty good. x11rb provides a robust wrapper around libxcb with a fully Rust-based alternative, and wayland-rs is the same with Wayland.

Still, if you want to write pure Rust programs, there is one annoying dependency that nearly every Rust program has. Let’s take a basic smol-based program, written top-to-bottom in Rust. Or so I think. Let’s see what ldd says.

$ ldd linux-timerfd
        linux-vdso.so.1 (0x00007fff58fae000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe7681bb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe767e00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe7682d4000)

Blech, disgusting! Let’ go over each of those libraries:

linux-vdso is the vDSO, which is used to implement certain system calls that can be reasonably implemented in user space. This object is used to prevent needing to incur syscall overhead unless it’s needed.
libgcc provides implementations for certain operations, like floating point operations and exception handling. The “_s” stands for “shared”, since it is a shared library.
ld-linux-x86-64 is the dynamic linker runtime. It’s what holds everything together.
libc is the C standard library, which contains wrappers around every relevant system call as well as a handful of C-oriented routines.

We’re going to spend most of our time talking about libc.

What is libc?

libc wears a lot of hats. It provides an extensive library of functions that are useful for C programmers, like fopen and memchr. It also contains more platform-specific wrappers around OS-specific functionality, like kqueue. In addition, for many operating systems, it’s the only stable interface between the user space and the kernel.

This part is important. Usually system calls, special interrupt instructions, are used to tell the kernel to do something important. However, these system call interfaces are usually unstable and prone to rapid, undocumented changes. This means that anyone trying to access kernel functionality has to go through libc, even if they aren’t actually C. This isn’t a suggestion; Go tried to use direct system calls on macOS a while back and got burned by it. It turns out, when they say “unstable”, they actually do mean “will change in inconsistent, backwards-incompatible ways”.

There are two important exceptions to this. The first is Windows, which has its own Win32 API that its libc is just a thin-ish wrapper over. This wrapper is how C programs written for Linux can sometimes still be used on Windows if it just uses the portable parts of libc. For our purposes it isn’t important. The second exception is Linux, which actually does have a stable system-call interface. You can call it from anywhere without going through libc, and you don’t have to worry about the actual calls changing out from under you.

Musl Melee

Because the userspace interface for Linux is the system calls and not the libc, you actually have a choice in what implementation of libc to use aside from “whatever the OS developer wants you to use”. There are two prominent implementations:

The GNU C Library (glibc), which is the battle-tested full-featured implementation.
Musl libc, which aims to be a simpler implementation focused on static linking.

In fact, if you were wondering what the “gnu” at the end of “x86_64-unknown-linux-gnu” means, it stands for the GNU C Library that Rust is using as an interface to the system. If we have the x86_64-unknown-linux-musl target installed, we can switch that out for Musl pretty easily.

$ rustup target add x86_64-unknown-linux-musl
$ cargo build --target x86_64-unknown-linux-musl

I wonder what ldd says instead now?

$ ldd linux-timerfd
        statically linked

Well, look at that! By default, the *-musl option automatically statically links the binary. No more dependencies! Everything is good, forever!

Rustix Revelation

Hmm, there’s something gnawing at the edge of my mind, like there’s something still wrong with this program. I just can’t put my finger on it.

It must be that there’s still C in there. Even though it’s statically linked, Musl is still a massive blob of unsafe, unsound, filthy C code.

Well, actually, Musl, and glibc for that matter, are extraordinarily well tested. Being that it’s a C standard library, it’s actually held to a very high standard for security and soundness.

But after my exposure to the Rustonomicon, my sanity has begun to decay like a flower wilting in winter. So let’s leave the realm of rational thought and imagine what can be. A veil lifts in my mind, revealing the Platonic ideal of a perfect program:

All Rust. Down to the very last bit. Perfect, clean Rust.

If only there was some way that we could tear that C code out by the teeth and leave it rotting at the wayside. But alas, our program needs that little bit of C code to run. Even if we were to rewrite our Rust code to use only syscalls, there would still need to be some glue code for program initialization, signal handling and threading. All written in Assembly and dirty, dirty C.

No, I see something wondrous. The answer to my prayers. What will purify my unclean executables and let them ascend into His Light!

The answer is eyra.

eyra is a set of libraries that aim to replace the role of the traditional libc in modern programs. It is written entirely in Rust, not counting the bits of Assembly necessary to tie the entire thing together. Not even a trace of C.

eyra was written by Dan Gohman, who is also the primary author of rustix. rustix is a safe wrapper around either raw system calls on Linux or libc on other platforms, and is a very fascinating piece of software that deserves its own blogpost. The point is, eyra is rustix taken to its logical conclusion: a complete replacement for libc.

The main drawback of eyra is that the process of integrating it into your program is more involved than just setting --target. But, it’s not so bad. Let’s write an eyra example program.

Enabling Eyra

First things first, let’s let cargo take care of scaffolding for us.

$ cargo new --bin eyra-example
     Created binary (application) `eyra-example` package

Let’s make it a little bit more complex than a “Hello, world!” program. Say, a smol-based TCP server that tells bad jokes.

$ cargo add smol fastrand eyre  
    Updating crates.io index
      Adding smol v1.3.0 to dependencies.
      Adding fastrand v2.0.1 to dependencies.
             Features:
             + alloc
             + std
             - getrandom
             - js
      Adding eyre v0.6.8 to dependencies.
             Features:
             + auto-install
             + track-caller
             - pyo3
    Updating crates.io index

In src/main.rs, we write a simple TCP server:

// in src/main.rs
use eyre::Result;
use smol::io::BufReader;
use smol::net::{TcpListener, TcpStream};
use smol::prelude::*;

const BAD_JOKES: &[&str] = &[
    "What do you call a fly without wings? A walk.",
    "Did you hear about the dull pencil? It was pointless.",
    "Why did the golfer bring two pairs of pants? In case he got a hole in one."
];

/// Handle an incoming connection.
async fn handle_connection(stream: TcpStream) -> Result<()> {
    // Wrap the stream in a BufReader to ease reading lines.
    let mut stream = BufReader::new(stream);

    // Read a line from the stream.
    let mut command = String::new();
    stream.read_line(&mut command).await?;

    // Remove the newline at the end if there is one.
    if command.ends_with('\n') {
        command.pop();
    }

    // Send a joke if the user asked for one.
    command.make_ascii_lowercase();
    if command == "tell me a joke" {
        // Choose a joke and send it.
        let joke = format!("{}\n", fastrand::choice(BAD_JOKES).unwrap());
        stream.get_mut().write_all(joke.as_bytes()).await?;
    } else {
        // Otherwise, send an error message.
        let message = "I only know how to tell jokes.\n";
        stream.get_mut().write_all(message.as_bytes()).await?;
    }

    Ok(())
}

fn main() -> eyre::Result<()> {
    smol::block_on(async {
        // Listen on a random port.
        let listener = TcpListener::bind("127.0.0.1:0").await?;
        let addr = listener.local_addr()?;
        println!("Listening at address {:?}", addr);

        // Start running an executor.
        let ex = smol::Executor::new(); 
        ex.run(async {
            loop {
                // Accept a new connection.
                let (stream, _) = listener.accept().await?;

                // Spawn a task to handle the connection.
                ex.spawn(async move {
                    // If an error occurs while running the task, print it.
                    if let Err(e) = handle_connection(stream).await {
                        eprintln!("An error occurred: {}", e);
                    }
                }).detach();
            } 
        }).await
    })
}

See the comments for a breakdown of how the program works, for those unfamiliar with smol’s API.

When we run the program, it tells us the IP address that it’s listening on:

$ cargo run
   Compiling eyra-example v0.1.0 (/home/jtnunley/Programming/eyra-example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.64s
     Running `/home/jtnunley/Programming/CargoTarget/debug/eyra-example`
Listening at address 127.0.0.1:44439

In lieu of a dedicated client, we can use netcat to test out the server.

$ echo "tell me a joke" | nc 127.0.0.1 44439
Why did the golfer bring two pairs of pants? In case he got a hole in one.

By checking with ldd, we can see that we’ve compiled this program against glibc.

$ ldd /home/jtnunley/Programming/CargoTarget/debug/eyra-example
        linux-vdso.so.1 (0x00007ffe8334d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4bb8376000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4bb8000000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4bb850a000)

Let’s try to integrate with eyra! (Not eyre, which I use to simplify error handling.) First, we need to add the latest version of eyra to our project. Let’s also add logging so we can see what eyra is doing under the hood.

$ cargo add eyra -F log,env_logger
    Updating crates.io index
      Adding eyra v0.15.2 to dependencies.
             Features:
             + env_logger
             + log
             - experimental-relocate
             - max_level_off

Using cargo tree, we can see that this pulls in c-gull, c-scape and a bunch of other things.

$ cargo tree
eyra-example v0.1.0 (/home/jtnunley/Programming/eyra-example)
├── eyra v0.15.2
│   └── c-gull v0.15.3
│       ├── c-scape v0.15.3
│       │   ├── < like, a lot of packages >
< snip rest out the output >

Then, we add extern crate eyra to the top of the main.rs file so that Rust knows to link to eyra, even if we don’t directly use anything from it.

// in src/main.rs
extern crate eyra;

// <snip rest of file>

Finally, we have to add a build.rs file, which is a little build script that runs before your Rust crate is compiled. We ue this to tell Rust to link using the -nostartfiles argument, which tells Rust not to bring in any of the C runtime. This is because eyra has its own initializing runtime, written in Rust.

// in build.rs
fn main() {
    println!("cargo:rustc-link-arg=-nostartfiles");
}

Now, we can run cargo build, which builds a significantly greater number of dependencies. Afterwards, we still have the eyra-example executable. Let’s see what’s inside.

$ ldd /home/jtnunley/Programming/CargoTarget/debug/eyra-example
        statically linked

Nice! It’s been statically linked, hopefully with 100% Rust code. Let’s run the executable with RUST_LOG=trace and see how it works.

$ RUST_LOG=trace cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `/home/jtnunley/Programming/CargoTarget/debug/eyra-example`
[TRACE origin::program] Program started
[TRACE origin::thread] Main Thread[Pid(89539)] initialized
[TRACE origin::program] Calling `.init_array`-registered function `0x563e1d8a5600(1, 0x7ffc4fff5d78, 0x7ffc4fff5d88)`
[TRACE origin::program] Calling `origin_main(1, 0x7ffc4fff5d78, 0x7ffc4fff5d88)`
[TRACE async_io::driver] block_on()
[TRACE origin::thread] Thread[Pid(89539)] launched thread Thread[89541] with stack_size=2097152 and guard_size=16384
[TRACE origin::thread] Thread[89541] marked as detached by Thread[Pid(89539)]
[TRACE polling::epoll] add: epoll_fd=4, fd=6, ev=Event { key: 18446744073709551615, readable: false, writable: false }
[TRACE polling::epoll] add: epoll_fd=4, fd=5, ev=Event { key: 18446744073709551615, readable: true, writable: false }
< snip: lots of logs from smol being initialized >

Let’s break it down:

[TRACE origin::program] Program started
[TRACE origin::thread] Main Thread[Pid(89539)] initialized
[TRACE origin::program] Calling `.init_array`-registered function `0x563e1d8a5600(1, 0x7ffc4fff5d78, 0x7ffc4fff5d88)`
[TRACE origin::program] Calling `origin_main(1, 0x7ffc4fff5d78, 0x7ffc4fff5d88)`

These logs come from the program starting up and setting everything up. It initializes the main threads, calls the program constructors (see the ctor crate if you want to know more about that), and launches the program’s entry point, origin_main.

[TRACE async_io::driver] block_on()
[TRACE origin::thread] Thread[Pid(89539)] launched thread Thread[89541] with stack_size=2097152 and guard_size=16384
[TRACE origin::thread] Thread[89541] marked as detached by Thread[Pid(89539)]

async-io, smol’s I/O driver, works by spawning a thread and then running epoll from that. This is used to deliver events throughout the program. Here we can see the driver being started, then a thread being launched to run epoll on.

As we can see, our program is now running on pure-Rust (and a little assembly) software. Does it work?

$ echo "tell me a joke" | nc 127.0.0.1 37279
What do you call a fly without wings? A walk.

Works like a charm!

Final Tally

Although it’s certainly a neat project that’s treading a lot of new ground, I probably wouldn’t recommend using eyra for a production grade project. It’s still wet behind the ears and it doesn’t add much practical value to projects. Still, it’s cool to be able to say that my project is 100% Rust.

The setup is somewhat convoluted. It would be nice if there was some subcommand that set up eyra for a project temporarily, like cargo-hack does.

Also, eyra still doesn’t support every libc function. It’s a slow uphill battle. They are open for contributions if you’re missing something important.

Until then, I’m very excited for what eyra will bring for Rust programs in the future.

Why you might actually want async in your project

2023-09-09T00:00:00+00:00

There is a common sentiment I’ve seen over and over in the Rust community that I think is ignorant at best and harmful at worst.

This blogpost is largely a response to this post by Matt Kline, but I’ve seen this kind of sentiment all over the Rust community. I’ve found that, in almost every case where async/await is mentioned, at least one person says this. It always gets on my nerves a little bit.

The guilty phrase is as follows: “async rust is only useful for a small number of programs, so why do library authors insist on using it in their APIs?”

I can understand where this kind of thinking comes from. Especially for newer Rustaceans, async/await is quite a bit of complexity up front. But, I think actively shying away from async is the wrong way to go.

Disclaimer: I am one of the maintainers for smol, a small and fast async runtime for Rust. So, obviously, I am somewhat biased. However, I think that async can be cool, small and fun; it’s just the presence of complicated async code used commonly across the ecosystem that scared people off.

Why async?

Greenspun’s tenth rule comes to mind quite often. For those unfamiliar, Greenspun’s tenth rule of programming is that every sufficiently complicated program contains a bug-ridden version of half of Common Lisp. Likewise, there are a worrying number of Rust programs that contain a bug-ridden version of half of an async runtime.

I call this “poor man’s async”. async/await is a natural pattern for doing multiple things at once; usually, non-async code tends to evolve into something closer and closer to async code, like carcinisation. It’s all over the place in the Rust ecosystem, once you start looking for it.

It happens like this: programs are naturally complicated. Even the simple, Unix-esque atomic programs can’t help but do two or three things at once. Okay, now you set it up so, instead of waiting on read or accept or whatnot, you register your file descriptors into poll and wait on that, then switching on the result of poll to figure out what you actually want to do.

Eventually, two or three sockets becomes a hundred, or even an unlimited amount. Guess it’s time to bring in epoll! Or, if you want to be cross-platform, it’s now time to write a wrapper around that, kqueue and, if you’re brave, IOCP.

Great, now you need a way to organize all of this work, because switching on it means you have to add two hundred lines to your program every time you want to handle some other corner case. No problem, let’s set up a queue of tasks. Whoops, turns out that queueing strategy is inefficient. Better make a new one. Let’s hope it’s thread safe!

At the end of this process, you’ve re-invented async-io and async-executor, two of the core components of smol.

This isn’t a knock on anyone in particular; this is a knock on me too! I’ve written quite a few Rust projects where I expect it to only involve blocking primitives, only to find out that, actually, I’m starting to do a lot of things at once, guess I’d better use async. The original async setup at the beginning of the project is somewhat annoying, but it’s a walk in the park compared to going back and rewriting my entire program setup to use async/await.

The point being, many people say that only five percent of Rust projects use async, and the remaining ninety-five percent have to put up with it. I disagree. Many of the remaining ninety-five percent (if that is an accurate number) are currently using async/await; they just haven’t admitted it yet!

Why don’t people like async?

Now here’s where I speculate why this attitude is so pervasive in the Rust community. I personally think it’s a combination of poor advertising on the part of async combined with poor standard library support.

I’d argue that, if you walked up to your average Rustacean on the street and asked what they thought of async programming, they’d argue that it’s just a obtuse, niche way to create web servers.

That’s not true! Even if you ignore the benefits of using async from a network clients, you can still definitely use async for desktop apps. async-winit is my attempt at bringing async/await to desktop apps in a managable way. I just think that too many people see async/await as part of Rust’s whole web shindig, instead of a reasonable way to structure highly concurrent applications.

In addition, the standard library is definitely built around synchronous code first and async code second. This means that a lot of async code that should be in the standard library ends up being pushed into external crates. This is definitely a problem that, thankfully, is being fixed as traits like Stream are now finally making their way into the standard library.

Even if you’re writing one of the simple programs that doesn’t explicitly need async/await, it’s not too difficult to move between the two worlds. Function colors get brought up a lot in this area of debate. However, personally, I find it much easier to go from async to sync and vice versa than JavaScript does. For instance, to run an async function in synchronous code, you can bring in the zero-dependency pollster crate and run this:

use pollster::FutureExt as _;

async { "Hello world!" }.block_on();

Likewise, to run sync code in the async world, it’s usually easy to spawn it onto a blocking task and then poll it from async code. There’s some thread-safety subtlety I’m papering over here, but overall it looks something like this:

blocking::unblock(|| "Hello world!").await;

Another benefit of async/await that I don’t know how to bring up organically above: it translates a lot better to web targets. Blocking synchronous code isn’t allowed in WASM in browsers, so by using async code, you can be reasonably sure that your algorithm can be ported to the web very easily.

Generally, async/await makes things more portable and easier to work into different application setups. I think that, if more Rustaceans invested the effort into learning how async/await ticks, we’d see it used in much more programs.

Keeping the Faith

I’ve been dancing around it for too long, let’s finally dive into this post.

The main complaint that the author has around async/await is that it requires your futures to be Send and 'static. This property tends to spread throughout the program.

async fn foo(&BIG_GLOBAL_STATIC_REF_OR_SIMILAR_HORROR, sendable_chungus.clone())

Except, this isn’t a problem with Rust’s async, it’s a problem with tokio. tokio uses a 'static, threaded runtime that has its benefits but requires its futures to be Send and 'static. In smol, on the other hand, it’s perfectly possible to pass around things by reference.

let big = /* ... */;
let chungus = /* ... */;

// With smol, you can create an executor...
let ex = smol::Executor::new();

// ...and, as long as its captured variables outlive it, you can pass things around from the stack!
ex.spawn(async {
    async fn foo(&big, &chungus).await
}).detach();

Actually, the main draw here is that this particular executor isn’t multithreaded. But it’s very easy to make it multithreaded.

// Create an executor.
let ex = smol::Executor::new();

// Create a channel used to stop the threadpool.
let (signal, shutdown) = smol::channel::bounded::<()>(1);

// Create a threadpool to run this executor on.
std::thread::scope(|s| {
    // Spawn 4 worker threads.
    for _ in 0..4 {
        let shutdown = shutdown.clone();
        let ex = &ex;
        s.spawn(move || smol::block_on(ex.run(shutdown)));
    }

    // Run a future on this executor.
    smol::block_on(ex.spawn(async {
        // Variables can be passed along just like before!
        async fn foo(&big, &chungus).await
    }));
});

Let’s say you don’t even want to use threads. You’re a fan of RefCell and Rc so thread-safety doesn’t really fit your use-case. That’s okay too! smol::LocalExecutor doesn’t require anything to be Send at all.

let my_thing = RefCell::new(4);
let ex = smol::LocalExecutor::new();

// Look, ma! A thread-unsafe task!
ex.spawn(async {
    *ex.borrow_mut() = 5;
}).detach();

Really, Send and 'static are not intrinsic properties of async Rust; it’s just what the biggest runtime decided on. If you’re not a fan of that, consider taking smol for a spin!

Wrap it up

Really, I think that most Rustacean’s fears of async are unjustified. Yes, there are complicated semantics at play, but really no more complicated than, say, a borrow checker. In exchange, you gain access to much more powerful program semantics (that you’re probably trying to use anyways!)

So, consider using async/await today! Even if you’ve been turned of by it in the past, there may be parts of the ecosystem that fit your use cases better.

What to expect from smol 2.0

2023-08-22T00:00:00+00:00

The tree of robust software must be refreshed from time to time with the blood of breaking changes. smol is no exception.

smol version 1.0 was originally released on September 7th, 2020. Since then, we’ve seen almost thirty new minor version bumps of Rust and a small handful of ecosystem changes. Some of these changes are so important that it necessitates changes to the way that smol is built.

In addition to hype building, the purpose of this blog post is to outline the changes that will be coming in smol v2.0. If you are a smol user, you should be ready for these changes when they come. The smol-rs team is planning on releasing smol v2.0 in the near future, although we don’t have a concrete date yet.

I/O Safety

Rust v1.63 was released with a new landmark feature: I/O safety. Prior to I/O safety, I/O resources were handled by passing around raw file descriptors and hoping that they were valid file resources. Here’s how a safe interface to the read syscall would be defined without I/O safety.

fn read(fd: RawFd, buf: &mut [u8]) -> io::Result<usize>;

In theory, this all works out fine. In practice, it is very possible for the file descriptor to be taken out from under you, through other threads or processes. Being ready for this essentially makes it impossible to write safe code that uses I/O resources.

I/O safety fixes this by assigning a lifetime to the file descriptor and guaranteeing that the underlying I/O resource will be valid for that lifetime. This shifts the burden of keeping those resources alive onto the borrow checker. The above function would be rewritten as follows:

fn read(fd: BorrowedFd<'_>, buf: &mut [u8]) -> io::Result<usize>;

I/O safety is very important to smol. In order to handle async I/O, I/O resources are registered into a global reactor. It is possible for the resources to be deallocated while they are still in this global reactor, which can lead to missing or spurious events. Now that we can assign a lifetime to the file descriptor before we register it into the reactor, this isn’t a problem anymore. smol v2.0 will have its Async type take AsFd types instead of AsRawFd.

This new advantage comes with a catch. Previously, Async allowed for easy &mut access to the inner I/O resource. However, this isn’t allowed now, as it is possible to move out and then drop the underlying I/O resource while it is still registered into the reactor, which voids the entire point. Therefore mutable access through methods like get_mut, readable_with_mut or writable_with_mut are now unsafe, with the condition that the user is not allowed to move out or drop the underlying file descriptor.

Unfortunately this change is very harsh and will likely break a lot of code in the wild. I’ve seen many real-world use cases use readable_with_mut for efficiently polling some I/O resource. Not to mention, the only real fix that can be done while still ensuring I/O safety holds is to use interior mutability, which is a code smell.

In any case, it is recommended for users of async-io, especially ones that implement async wrappers around I/O resources like inotify, to expect this change and start thinking about how the architecture should be changed.

See this pull request for polling for more information on why this was done and how it was considered.

`async-channel` and `async-lock` !Unpin Futures

Previously, async-channel and async-lock (aka smol::channel and smol::lock) had several methods that returned Unpin futures. These futures can be polled without pinning them to the stack or heap. Once smol v2.0 releases, these futures will now be !Unpin.

I expect this to have a low impact on your average bear, as most of the time these futures are immediately .awaited, which doesn’t care about Unpin at all. However I am aware of some use cases that rely on the Unpin-ness of these futures. Fortunately this breakage can be ameliorated in the short term by just Box::pinning the future and polling it from there.

The reason for this change is that it allows the async primitives underlying these libraries to be implemented in a much more efficient way. These libraries work by storing waiters- tasks waiting for the channel to receive a value or the Mutex to unlock- in a linked list. Previously, this linked list used a strategy where every node in the list was heap allocated. Now, these nodes use stack storage when possible, which avoids allocations on the heap.

This optimization is a massive win for performance, doubling it in some cases. However, this strategy requires that the futures be !Unpin, as the stack storage cannot be moved without invalidating every other node in the linked list.

`async-channel` and `async-lock` on `no_std`

Previously, async-channel and async-lock were only available for platforms with std enabled. This is because they internally made use of std::sync::Mutex for synchronization. Once smol v2.0 released, these crates can now be used on no_std platforms.

All features from these crates should still be available on no_std platform, except for blocking methods (like send_blocking). The main drawback is that both of these crates still require a global allocator, which probably can’t be removed until allocator_api is stabilized. So unfortunately it probably won’t be used on embedded systems any time soon.

Once again this win comes from the underlying event notification primitive. The previous implementation used an std::sync::Mutex to synchronize the state of the inner linked list of wakers. Although the std implementation still uses this mutex, no_std uses another strategy. The state is protected by a spinlock, but put away your blogpost links for now. Only a limited amount of spinning is allowed, and it falls back to a fully atomic queue after the spinlock fails too many times. This strategy was based on analysis that the spinlock is the most efficient strategy for low contention, and the atomic queue can act as a fallback for when the contention is high.

However, the no_std strategy is significantly slower than the std strategy. Therefore users should use the std strategy whenever possible. Users of these crates should consider exposing an std default feature in turn that allows for users to opt-out of using std for no_std applications.

Although this win and the previous one were developed over a long series of pull requests, this one is the most relevant.

`futures-lite` now re-exports `ready` and `pending` from libstd

When futures-lite was originally written, ready and pending were not available on libstd yet. However, as of Rust v1.63, they can be imported from core::future. Therefore, futures-lite will now re-export these functions from core::future instead of defining them itself.

This is a small but significant win, as it represents less functionality needed on the part of futures-lite. However, our v1.63 MSRV stops us from re-exporting poll_fn as well, which was introduced in v1.64.

See this PR for more information.

ESP-IDF Support

Thanks to Ivan Markov, the underlying machinery of smol now has better support for ESP-IDF. This is a big win for the embedded community, as it allows for smol to be used on embedded systems that use ESP-IDF.

New Logo

We now have a new logo! The old smol logo was a smol, cute orange tabby cat. To represent the new capabilities that smol has for version 2.0, we’ve decided to change the logo into a cyborg cat!

Thanks to NebulousStar on X for drawing this!

…and much more!

There are a handful of other minor, non-breaking changes as well. This includes the task system now having a metadata slot available, and the async-executor crate using thread-local queue optimizations. These changes are not as important as the ones listed above, but they are still worth mentioning.

I’ll have a more complete changelog once the release actually takes out. Until then, you would be well advised to start thinking about how these changes will affect your code.

Announcing Theo

2023-08-04T00:00:00+00:00

Today I’m happy to officially announce a project that has been the culmination of about five months of work. This makes it probably the most involved programming project I’ve ever done, and I’m proud of the final result.

The GUI ecosystem in Rust is still in a nascent stage. There are well-defined ecosystem crates for doing certain things, like window management. On the other hand, other essential GUI components are still wanting for a good solution. In my own travels, I’ve found that there’s no good out-of-the-box solution to vector graphics yet, which is an essential part of any GUI. The closest thing is piet-common, but it’s hard to integrate into non-druid systems like winit.

theo is my attempt at resolving this issue. It’s an implementation of piet, a common vector graphics API, that can be used with any windowing system that implements the raw-window-handle traits. The goal is to provide a system that can be effortlessly plugged into whatever system you already have in place, and provide a simple, easy-to-use API for drawing vector graphics.

It has a number of benefits over piet-common and other contemporary solutions. First of all, as mentioned, it works with the raw-window-handle interoperability traits, meaning that it should, in theory, work anywhere. Another primary benefit is its use of GPU acceleration. Where possible, it uses either wgpu or OpenGL as the underlying drawing backend, which should provide benefits for systems with lower CPU power, like Risc-V boxes. However, it is able to fall back to a software rasterization backend based on tiny-skia and softbuffer, which should work on any system that can run winit.

theo also uses very few system libraries. piet-common brings in cairo on Linux. On the other hand, aside from GPU libraries like Vulkan and windowing libraries like X11, theo depends on very little.

If you’re building a GUI system, consider giving it a try! See the examples in the repository for an example of how to use it.

The Vellophant in the Room

At the time of writing, the Linebender team is working on a similar vector graphics system named vello. It uses GPU acceleration via wgpu and is designed to work with systems like winit as well. So why did I go to all the effort? Why not just wait for vello to be ready?

First of all, theo and vello use different strategies. vello is a compute-based renderer, while theo uses a rasterize renderer. The difference comes down to strategy: vello tries to use the GPU to the fullest extent by uploading vector data to the GPU and then rendering it there. On the other hand, theo uses a more traditional strategy: converting shapes on the CPU to textured triangles and then pushing those to the GPU. It is this author’s opinion that the rasterization-based strategy is more ergonomic and easier to cache, so I created theo around this strategy. Your mileage may vary.

There are some other minor differences, but overall once vello is actually released it should work similarly to theo. It’s a matter of API preference, really.

Was it hard to make?

Yes.

Other Crates

While building theo, I had to write a handful of other crates to make the current Rust rendering ecosystem easier to work with. Here’s a quick rundown of them:

line-straddler, which generates underlines and strikethroughs for glyphs.
piet-cosmic-text, which provides a uniform text rendering interface for piet using cosmic-text.
piet-tiny-skia, which implements a piet backend using tiny-skia.
piet-hardware, which converts piet rendering calls to buffering textured triangles.
piet-glow, which implements a piet backend using OpenGL.
piet-wgpu, which implements a piet backend using wgpu.

The Quest for Piet 3, or, Text Rendering Madness

2023-08-03T00:00:00+00:00

Whenever I’ve talked to someone else who writes a drawing interface like this, it seems like they always say text is the hardest part. But obviously, I’m built different, right?

Last time, I integrated gradients more completely into my piet-hardware crate. Today, we’re moving onto a subject that’s much more likely to be bad for my blood pressure: text rendering!

For the most part, I offload text loading, shaping and rendering onto the cosmic-text crate. It’s a pretty complete implementation of the whole “text” thing. Not to mention, it’s been created by people far smarter than I am.

So far, it’s worked well. In my limited tests, it’s created a decent layout system for text. The rendering has also worked out pretty well.

Unfortunately, especially when it comes to the GPU backend, there’s a lot of things that we have to take care of. Principally, the piet text API needs to be translated to cosmic-text calls. I’ve created a crate to handle this translation: piet-cosmic-text. But I’ve long suspected that something is wrong.

There’s also the fact that we need to draw the text onto the window. By default, cosmic-text draws to an array of pixels, but that doesn’t translate well to OpenGL. What I do is render those pixels to a texture, and then draw portions of that texture to the screen. This technique is called texture atlassing, and it’s very common when using this kind of drawing code.

So! Without further ado, let’s find out what’s gone wrong.

Number 5: Coloring Cascade

The purpose of this one, I think, is to test how text rendering deals with multiple colors and styles in one block of text. Here’s the reference image from piet-cairo:

Here’s what piet-glow and piet-wgpu spit out:

Right! A few issues I can spot right off the bat:

The text is too big.
The text stops being blue way too soon.
The font is too bold throughout.

The first one seems like the easiest to tackle. In the code for piet-cosmic-text, there’s a function for converting the piet font size to the cosmic-text font size:

fn points_to_pixels(points: f64) -> f64 {
    points * 96.0 / 72.0
}

This seems to be making it too big, so maybe I remove the ratio?

fn points_to_pixels(points: f64) -> f64 {
    points
}

This seems to go the other way; the output is now too small?

It looks like the ratio depends on the DPI of the screen, which varies from place to place. If you don’t have access to that information, you should assume that it’s 96 DPI. So, what I was doing originally was right?

To me it seems like this should be an adjustable option on the part of the user. The windowing framework is generally what controls the DPI, so it should be able to tell the drawing code what it is. So far winit doesn’t support a straight way to fetch the DPI (it exclusively uses scaled pixels, which doesn’t really help us here). So I’ll just keep it ratio-free for now, since that seems to get us reasonably close to what we want.

Anyways, the next item on the list: the text stops being blue. I think this is because I mistranslate the piet attributes to cosmic-text attributes. There’s no real excuse here outside of me just misunderstanding the math. After rewriting the algorithm I used to choose text attributes, everything seems to be fixed… except for one thing.

The underlines and the strikethroughs are in the wrong places. I use a crate called line-straddler to figure out where the underline should go, but it seems to be off.

What kind of hack designed this broken crate? I oughta- oh, it’s me. I wrote line-straddler. Whoops.

One rewrite later…

Oh whoops, the lines are too skinny. Easy enough fix.

No, Kronk! The other lever!

Okay, pros: the lines are now at an appropriate thickness! Cons: the lines put themselves just about wherever they please.

Why is it so hard to draw lines under text?

Let’s- wait a second, cosmic-text v0.9.0 just released with breaking changes. Normally I’d complain about this happening right in the middle of what I’m doing, but it removes the bespoke ouroboros dependency, so I’m a happy camper.

After all of that and a little math adjustment, I get this:

Looks good, but a few small issues:

Why is the text justified all of a sudden?
The underlines are too thick.
The lines start from the top down instead of the bottom up like they do in the cairo implementation.

Issues 2 and 3 are a matter of changing a few lines of math. Issue 1 looks to be an actual cosmic-text bug introduced in v0.9.0. Let’s file that. Until then…

Close enough for government work. The bug seems to be localized to when fonts are switched mid-paragraph, so I’ll just avoid doing that for now.

Holy hell, that was an ordeal. Good God, please give me something easy next…

Number 7: Alignment Ailment

This test aims to test how text alignment works, for both left-to-right and right-to-left text.

Here is the sample version, from piet-cairo:

Mhmm, seems like there is some overlapping text. Let’s see how we did…

Great! We’re halfway there already, thanks to all the work we did on the last sample. The only thing left to do is to fix the right-to-left text at the bottom.

cosmic-text doesn’t have an option out of the box. Implementing it doesn’t look too hard, though. That’s a PR!

It’s been merged, but it’s not in release yet. So I guess we’ll just wait for that.

Number 8: Font Size Frenzy

The catch with this one appears to be testing multiple different font sizes. There’s also different colors and fonts, but we’ve already tested those.

Here’s what it looks like under piet-cairo:

And here’s what it looks like under piet-glow:

…nothing! That’s right, nothing at all! It throws an error and outputs nothing!

This is because cosmic-text doesn’t actually support multiple font sizes in one block of text. Apparently it’s harder to lay out like that. In fact, here’s the code I wrote to handle that:

TextAttribute::FontSize(_) => {
    // TODO: cosmic-text does not support variable sized text yet.
    // https://github.com/pop-os/cosmic-text/issues/64
    error!("cosmic-text does not support variable size fonts yet");
    return Err(Error::Unimplemented);
}

Bah, humbug! Who cares about one little sample. Gotta crack a few omelette to make an egg, right? Let’s just move on to the next sample.

Number 9: Unsizing Ultimatum

This one can’t be too bad, right? It’s just—

Oh, no. Those font sizes look a little bit variable, don’t they? That means that…

Yep! Nothing! That’s right, I got the exact same error!

So, I guess this means we have to deal with variable sized font in some way. One failing sample would be reasonable. Two would be criminal.

There’s a pull request that adds support for variable sized fonts, but it’s still in its early stages and touches a lot of code. I’d also like to avoid depending on git repositories for now.

Implementing it myself would require a lot of code to be written. Code that would be thrown away once the PR is merged and released.

A simple patch is just to remove the error and make it ignore the changes in font size. It’s not ideal, but for now it has to work.

Oh, the little red and green lines are all wrong, aren’t they?

I’ve taken a look at the code, and I think I’ve decoded what it means:

For each line of text, it returns line metrics.
The red line represents the “top” of the line.
The green line represents the “baseline”, where the bottom of most letters end.
The blue line represents the bottom of the line. I’m not sure why this only appears once?

So it’s only a matter of adjusting the metrics. Easy enough. Right now, my definition of metrics were somewhat off-the-cusp, mostly just to fill in the text rendering trait.

After reviewing the definitions of the LineMetrics struct and adjusting accordingly, here’s what I found:

Whoops, I added the line height to a couple of them by accident. Allow me to fix this:

There we are! The last line seems misplaced to me… but you know what, it’s close enough for government work.

For all those reading, in case you’ve wondered why there are hundreds of thousands of lines of code in a text rendering library, this is why. “Correct” text rendering is a very complicated subject.

Say, we never checked in on sample number eight after we “fixed” text sampling, did we? Before we move on, let’s check in on it.

Wait, is part of the text just missing? Grooooaaan

Revenge of the Eighth

Our text is missing! Why is is missing? Well, why not? Why not have text randomly disappear without as much as a warning log message? That seems sensible.

After some slight trial and error, it looks like if I disable the bolding on the latter portion of the text, it renders fine.

Maybe cosmic-text just doesn’t like it when you have a font with weird bolding like that? I don’t know, and from looking at the code it isn’t immediately obvious why this happens.

Why is this happening? Why isn’t the text showing up? Why isn’t there as much as a log message when this happens? Is it a bug? Is it a feature? What’s wrong with this project? Why have I poured so much time into this? Does everyone spend this much time on text rendering? What happened to Site-13?

The Long Haul

Seriously, what happened to Site-13? When it was originally released in 2015, it was something completely new for the SCP Foundation. It took the long exploration logs previously explored in articles like SCP-093 and really ran with it. The viscerally terrifying exploration logs were like something right out of a Raimi movie, and the sheer scale is still almost unmatched to this day.

But then, in 2018, the original author, djkaktus, added more to the story. He added a handful of exploration logs, followed by a mouthful of interviews. In my opinion, this is one of the worst mistakes in the history of the SCP canon. SCP-1730 already had a very complete arc to it, and these new logs unnecessarily prolonged it. Not to mention, the new logs didn’t really fit the tone all too well.

I don’t want to spoil it, but imagine if Evil Dead ended with an hour-long scene where a handful of cyborg ninjas shows up and beats up all of the Deadites, followed by another hour-long scene where they interview all of the teenagers to figure to what happened.

This tragedy has always been the prime example for me on why you need to reduce your scope. Making your projects too large is hazardous to both your health and your final quality. Even if you manage to finish your project (and for someone as easily distracted as me that’s a big if), what’s left will probably be a bloated mess that’s not worth the effort.

When I originally started getting into Rust in 2019, I identified that there wasn’t a real good solution to GUI in Rust. Therefore, I naively set out to single-handedly create an entire GUI framework, top to bottom, from scratch.

It took a year on flailing about in various aspects of GUI management to find out that this was a bad idea. A GUI framework touches on an absolutely massive number of subjects, from windowing systems to vector graphics to text rendering to fundamental data structures. That’s not even all of it; that’s just the underlying plumbing. When it comes to the parts of the GUI that matter, you need to come up with an incrementalization framework. Good luck doing it in a way that works with Rust’s object model and isn’t an absolute mess to use.

At this point I decide that scope reduction is a good idea. Okay, maybe I don’t want to write the entire windowing system. winit already exists and works well enough; I can just patch it until it fits all of my use cases. Maybe I don’t want to deal with making my own underlying graphics API. piet is in use by druid and it seems to be working for them. Actually, hmm, the default piet implementation, piet-common, doesn’t play well with winit.

Now, here’s a scope that I can shrink down to! It should be easy enough to just write a piet, I foolishly said to myself. Once I implement all of the piet traits, it’ll just be easy, right? Right?

So I put myself to work. The hardest part was writing the OpenGL-based implementation. I explicitly wrote it so that it would be portable to wgpu, so I could just swap out the backend and it would work. For text rendering I figured I could just throw cosmic-text at it Then I wrote a software rasterization backend so that it could work pretty much everywhere. It took a couple of months, but it seemed to work, right? I even used it in a school project of mine.

But then, bug reports started flowing in. Clipping wasn’t working properly, colors weren’t matching up quite right, and text rendering? Forget about it.

So in lieu of tracking all of these bugs down, I noticed that piet had a bunch of samples that it used to test itself. I figured that if I could get all of those samples to work, then I could be reasonably sure that my implementation was correct.

It wouldn’t take too long, right? Just a few little tweaks right? Since I’d already dogfooded it in a school project, it should be fine, right?

Well, you’ve been there for the rest of it.

Frankly, I’m not even sure I want to continue developing that GUI framework I’d planned after this. Not only is the GUI situation much better than it was back in 2019, but I think it’s just too much work for one person to do. Especially as I’ve graduated college now and entered full-time employment, I’m often lacking the time and energy to do it.

I’m not sure what to do at this impasse. I think that this package I’m writing, theo, will be a net positive for the Rust community, as something like it for winit doesn’t really exist yet. So I’ll at least finish it. Maybe that’s just the sunk cost fallacy speaking; I don’t know.

Anyways, we’ve got some text to fix.

Fixing cosmic-text’s Disappearing Font Problem

Anyways, I think I’ve figured out what the issue is, at least on the cosmic-text side.

In text rendering, there is a concept of “font fallback”. It first tries to load whatever font you requested. This loading keeps parameters like boldness, italics and font size in mind. If it can’t find the font you want, or if you’re rendering characters that aren’t in that font, it tries to find another //fallback// font. It starts off with fonts that look like the original font, and eventually starts moving onto whatever fonts it’s hardcoded into the system, just to get //something// onto the screen.

cosmic-text has a system like this, but it turns out that the fonts that it uses as a fallback are missing on my system. Hence, the missing text. Although it is annoying that it didn’t log anything as an error, there’s only one entry buried in the debug log.

I could “fix” this on my system by just installing the fonts, but I’d also like to fix this across all systems. Text not showing up is a pretty big deal, after all. So what if I embed the fonts that I need into the executable itself, so that you’re never without the font that you need to fall back on?

The strategy I ended up using was to download the DejaVu font collection. Then, in the build script I take these fonts, compress them using the Zlib compression scheme (which saved the most space in my experiments) and then put them in the output directory. piet-cosmic-text then uses include_bytes! to embed these fonts into the executable.

From there, it’s just a matter of telling cosmic-text to use these fonts as a fallback. Did that fix it?

No, it did not.

Apparently, the disappearing font comes not from the font fallback, but from shaping. After some review of cosmic-text’s source code, I think I’ve figured out the real issue. Apparently if the shaper fails to find the ideal glyph ID, it’ll try to shape it again but with another font. If it can’t find that glyph ID in any of the fonts, it’ll just give up and not render anything.

So I set up a loop that checks each rendered glyph to see if they have a “glyph ID” of zero. If they do, it indicates that we need to change the ID for the better.

Okay, we’re almost there now. It’s justified again (not my fault) and the spacing between the lines is weirdly bolded (is my fault). A simple patch fixes it.

Close enough for me, for now. The font size is still off, but that’s a job for a later version of myself.

Sample 10: Font Decoration Frolic

Alright, this has gone on for long enough, and had far too many tangents. This sample aims to demonstrate how text decorations work, especially with Zalgo text. Please, let this be an easy one.

Here’s the reference image from piet-cairo:

Here’s what piet-glow spits out:

Augh! So close! Aside from being unnecessarily blurry (and maybe rendering a couple of the decorations wrong, but I figure, close enough), it seems that the bounding box of the text is wrong. The white rectangle containing the text should be smaller.

Apparently the black outer rectangle corresponds to the “ink rectangle”, which represents the total extent of the text. The rectangle should go around everywhere the text touches. Meanwhile, the white text is the “logical extent”. Remember when you were writing text between the lines in middle school, before they all switched to using Chromebooks for everything? The logical extent would go from the top line to the bottom line.

The logical extent is relatively easy; hell, when we calculated the line metrics above, we basically already got that. The ink metrics are the harder part, because it’s hard to tell from the parsed font alone. For vector graphics you can figure out the bounding box, and we can take a similar strategy from the raster image.

Wow, not even close.

Actually, you know what? We’re long past the point of “doing things right”. You know what the bulletproof way of doing this is? Actually rendering the text, then taking the bounding box of that.

Here’s the candidate for the worst code I’ve written so far:

let mut ink_rectangle = Rect::new(f64::MAX, f64::MAX, f64::MIN, f64::MIN);
buffer.draw(font_system, &mut ct::SwashCache::new(), ct::Color(0), |x, y, w, h, _| {
    ink_rectangle.x0 = min(ink_rectangle.x0, x as f64);
    ink_rectangle.y0 = min(ink_rectangle.y0, y as f64);
    ink_rectangle.x1 = max(ink_rectangle.x1, x as f64 + w as f64);
    ink_rectangle.y1 = max(ink_rectangle.y1, y as f64 + h as f64);
});

Now, I’m not proud of this code. It basically goes through the entire rendering process and created a new cache just to calculate the ink rectangle. Let’s clean it up a little:

Rather than using Buffer::draw, I use swash::Scaler, which is how swash calculates the bounding boxes of the text in the first place. Unfortunately a small amount of rendering takes place, but that’s okay. It puts a permanent dependency on swash for piet-cosmic-text, which may make it harder to swap out the renderer for downstream crates.
Rather than creating a new SwashCache every time, the shaping context is cached inside of the Text handle.
We use a hash map to store the bounding boxes of each glyph, so that we don’t have to calculate them every time.

After all of that, we get a decent enough performance boost, followed by a decent enough result:

If anything, I’d argue that it’s more accurate than piet-cairo’s sample.

Conclusion

Well, this has been a roller coaster so far, and it’s still far from over. This isn’t even the last we’ll see of text rendering, but I think I’ll try to take a break from it for now. Next time, we’ll see if we can figure out the rest of the samples.

notgull

SCP-093 is a Timeless Masterpiece

Context

SCProcedures

Description

Original Documentation

Test Log 1

Test Log 2

Blue Test

Green Test

Violet Test

Yellow Test

Red Test

Recovered Materials

Speculation

Parting Shots

The rabbit hole of unsafe Rust bugs

Event Listener Escapade

Debugging Demonology

Damaged Dependencies

Goose Chase Gallery

miri Magic

Arc Apocalypse

Parting Shots

Evaluating new software forges

Software Host Hellscape

GitLab Gauss

SourceHut Scramble

Codeberg

Self-Hosting Gitea

Parting Shots

My new, remote-access Rust development setup

Time for an Upgrade

The Hardware Specs Section

Putting It All Together

Utilizing the System

Desktop Dismay

Conclusions

Recreating concurrent futures combinators in smol

The Problem with futures

Concurrency Conundrum

Make your own, better combinator

Parting Shots

Eyra is an interesting Rust project

What is libc?

Musl Melee

Rustix Revelation

Enabling Eyra

Final Tally

Why you might actually want async in your project

Why async?

Why don’t people like async?

Keeping the Faith

Wrap it up

What to expect from smol 2.0

I/O Safety

async-channel and async-lock !Unpin Futures

async-channel and async-lock on no_std

futures-lite now re-exports ready and pending from libstd

ESP-IDF Support

New Logo

…and much more!

Announcing Theo

The Vellophant in the Room

Was it hard to make?

Other Crates

The Quest for Piet 3, or, Text Rendering Madness

Number 5: Coloring Cascade

Number 7: Alignment Ailment

Number 8: Font Size Frenzy

Number 9: Unsizing Ultimatum

Revenge of the Eighth

The Long Haul

Fixing cosmic-text’s Disappearing Font Problem

Sample 10: Font Decoration Frolic

Conclusion

`miri` Magic

The Problem with `futures`

`async-channel` and `async-lock` !Unpin Futures

`async-channel` and `async-lock` on `no_std`

`futures-lite` now re-exports `ready` and `pending` from libstd