PDF files, and why the heck they are so slow to read. Hours upon hours of perf(1) and fiddling with ugly things in C. My main takeaway is everyone in the world is doing things HORRIBLY wrong and there's no way to stop them.
(Digression: did you know libpng, the one everyone uses, is not supposed to be an optimized production library—rather it's a reference implementation? It's almost completely unoptimized, no really, take a look, anywhere in the codebase. Critical hot loops are 15 year old C that doesn't autovectorize. I easily got a 200% speedup with a 30-line patch, on something I cared about (their decoding of 1-bit bilevel to RGBA). I'm using that modified libpng right now. I know of nowhere to submit this patch. Why the heck is everyone using libpng?)
The worst offender (so far) is the JBIG2 format (several major libraries, including jbig2dec), a very popular format that gets EXTREMELY high compression ratios on bilevel images of types typical to scanned pdfs. But: it's also a format that's pretty slow to decompress—not something you want in a UI loop, like a PDF reader is! And, there's no way around that—if you look at the hot loop, which is arithmetic coding, it's a mess of highly branchy code that's purely serial and cannot be thread- nor SIMD- parallelized. (Standardized in 2000, so it wasn't an obvious downside then). I want to try to deep-dive into this one (as best as my limited skill allows), but I think it's unlikely there's any low-hanging optimization fruit, like there's so much of in libpng. It's all wrong that everyone's using this slow, non-optimizable compression format in PDF's today, but, no one really cares. Everyone's doing things wrong and there is no way to stop them.
Another observation: lots of people create PDF's at print-quality pixel density that's useless for screens, and greatly increases rendering latency. Does JBIG2 support interlacing or progressive decoding, to sidestep this challenge? Of course it doesn't.
Everyone's doing PDF things wrong and there is no way under the blue sky to make them stop.
For me, it's DIY audio.
The thing about diy'ing audio (primarily speakers but also amps, DACs etc) is that you can get top of the line performance for a fraction of the market price. A $50,000 speaker setup that would bring tears to your eyes could be made for perhaps $5000. A DIY $500 kit can perform similar to a $2-3000 set of speakers. Open source amps with gerber files on github are amazing.
The biggest reason it's so easy to get amazing value is because that $600 speaker only has $150 of materials. Upgrading its $25 woofer to a $80 one would help a lot, but no company would do that and not sell it now for $1000 if they could.
However the biggest allure for me is not beating commercial systems on cost, but making what I want. A small speaker with deep base? Easy. Speakers with quasi-active noise cancellation behind them? Sure, why not. Speakers that'll make the most overpowered/fancy beach-boombox sound like a crappy toy? Simple.
The only limit is your imagination and time/money.
I'd very much recommend diyaudio.com, but be warned, parts of this field are mature while others are still in effective infancy. Also, being an engineer (electrical/mechanical) helps a lot, there's a ton of signals processing and electrical/mech oscillation.
I did not realize that learning Fusion 360 was going to be such a huge chapter in my current journey. Looking back, I'm kind of stumped at how I avoided it as long as I did.
I would now put learning CAD in the same category of mandatory life skill as learning to code. The ability to translate what you see in your mind to something that can be repeatably fabricated is an incredible power move, akin to learning how to communicate complex ideas with empathetic language.
My advice is to start by following this tutorial step-by-step. It's a 90 minute video that took me ten days to get through. Step two is to take an existing project and change it in a significant way. Step three is to create something from scratch which solves a problem that you have.
Self-hosting and homeserver stuff!
I started out when I got a new phone, and didn't know what to do with the old one. One of the ideas I had was a homeserver. Turns out it's not trivial to run Docker even on rooted Android phones, and you need a lot of kernel patching, tweaking and more and it still had issues after that.
The next step was when I figured out I could install postmarketOS on it, and I managed to flash it, SSH into it and set up Nextcloud for our photos and unbound as a recursive DNS for my home network. I thoroughly recommend postmarketOS, and the contributors are amazing as well.
I was however running out of storage, so I ordered an 256GB SD card, and set up mergerFS between it and local storage, worked fine.
After some time however, I got paranoid about having and old device with a LiPo battery constantly being charged in my home, so I decided to get a mini PC from Aliexpress and chucked a 2TB SSD in.
In the meantime, I discovered Immich, which turned out to be much better for photos than Nextcloud, and fell in love with it.
The final thing I added was a miniDLNA service to play my local movies and shows on my LG TV without having to bother with Plex/Jellyfin and reencode anything. Unfortunately, it kept disappearing after roughly 2 days of operation, so I just added a cron job to restart it at 5 AM.
For the time being, I don't need anything more and am turning my attention to other things.
I’ve fallen into the plant tissue culturing rabbit hole. I was selling excess trimmings of aquarium plants locally on Facebook marketplace and made a surprising amount of money, and my kids really enjoyed it (they got a cut for helping me out). I thought hmm, this could be a great excuse to make a little business around this, teach them some skills, get them thinking more constructively and feeling a sense of agency and ability, etc. Plus earning money is really nice when you’re a teenager.
The challenge is that in an aquarium, plants grow reasonably fast but not fast enough to sell regularly for a decent income. You need ways to produce more plants faster, more reliably, and without taking up too much space. That’s where tissue culturing comes in.
It has reaaaally sucked me in. I’m culturing everything I can find. I’m also propagating aquatic plants through more typical means, and that’s fun too. I’m out of space though.
Tissue culturing is a really fascinating science and practice. I love keeping track of the media recipes, results, growth rates, etc. I’m too early to have had meaningful results, but I look forward to tracking those as well.
I've been attempting to relearn music theory for the third time.
The first attempt was over a decade ago, and I never really quite got it.
The second attempt was about 6 years ago, and while the fundamentals clicked and I improved dramatically as a musician/composer, I still only ever learned some small portion of the basics.
This time, I'm taking it a few steps further. I'm going back to the basics first to recheck my existing understanding of things, and really trying to take my time and understand each new concept before moving on.
I'm currently working my way through this online textbook: https://musictheory.pugetsound.edu/mt21c/frontmatter.html
I've also picked up a simple little budget keyboard (a 61-key Casio CT-S1) that does just what I need at a great price.
Having a lot of fun so far and learning a ton! :)
I deep-dove into the identity of a spammer. They were sending me the crappiest, absolutely lowest-denominator hodge-podge of spam. Fly repellent, ultrasonic dog whistles, knives. It all seemed like legit businesses, so it was just some marketing firm that bought an email list somewhere.
Whenever I get spammed, I check the email headers. Turns out they were using an email with the owner’s initials in the unsubscribe header. From it, I was able to easily guess their actual name. I found them on LinkedIn! I used that, plus the list of all the domains they used for marketing and sending mails, to build a pretty comprehensive map of their operations.
I thought for a while about what I could do with this info… but in the end, reporting them to my country’s consumer rights authority for spam did the trick. No reason to get in trouble myself, as fun as it could be.
So the lesson is: look at email headers! There’s fun stuff in there!
I started by looking into LLMs outside of ChatGPT (self-hosting ollama, API services, etc.) I got annoyed that they all had their own 'chat' interface, or that all the chat interfaces were some bad rebuild of a chat application.
So I wrote https://github.com/arcuru/chaz to connect any LLM to Matrix so that I could use any Matrix client with any LLM.
That got me writing a Matrix bot account in Rust, and I couldn't find a simple bot framework so I split out a bot framework in https://github.com/arcuru/headjack
But _then_ I realized a really easy bot I could write that would also be very useful for me, and I wrote https://github.com/arcuru/pokem which is sort of a clone of ntfy.sh but using Matrix. Send yourself (or others, or a group) a ping on Matrix using an HTTP request or a simple CLI app.
Micropython on an Adafruit Feature to make a dual deploy altimeter for high power rocketry. Got really into control theory and all the fancy math for determining the state of the rocket on the way up. Then got into device drivers and improving on the ones that come with CircuitPython to get higher performance out of barometers and IMUs. Then got into circuit board design and fabrication starting from zero for an i2c pyro board that fires the ejection charges (little tubes of blackpowder and an e-match used to eject parachutes). That was pretty interesting, i have the advantage of a handful of hardware engineer friends to answer my questions when i get stuck.
It's all sitting on my desk, first flight will likely be in May.
I recently fell into the Linux from Scratch rabbithole. The extra unexpected fun came while attempting to build LFS from within NixOS (my daily driver, which I only had a basic proficiency in). I quickly realized the challenges of following the LFS project's guidance, due in large part to how the Nix store is implemented. So I fell down another rabbithole of learning derivations and the Nix language - which took me a good way though building the LFS toolchain within Nix's intended sandbox. However, achieving FHS-compliance became another issue as I attempted to build essential LFS system tools within a chroot-like environment, but sought to do so in roughly the same declarative/reproducible manner as Nix's ideal. After a few lost hairs, I discovered the wonders of Nix's FHS build environment/bubblewrap implementation. A few hoop-jumps later, handling permissions and migrating to the final build partition, the project was complete with a mostly declarative, mostly reproducible, functional, minimal, bootable LFS build.
What sticks with me most through this experience is the brilliance of the open-source community (and a special satisfaction with now being able to say "I use nix btw")
I've been making milk punch for friends as a gift for years now. On a lark I wanted to figure out how to produce it in larger batches with less manual labor and discovered the tip of the iceberg of what is the field of beverage filtration and food chemistry.
Turns out getting particulates out of a solution is a massive, massive industry with a large body of science, literature, and engineering practice behind it.
EDIT: Here's a few wiki entries I found as OK overviews. ChatGPT was handy for figuring out what relevant literature in the field was and terminology I could use to find more pertinent resources:
1. https://en.wikipedia.org/wiki/Food_engineering
2. https://en.wikipedia.org/wiki/Ultrafiltration
3. Food Chemistry: https://www.amazon.com/Fennemas-Food-Chemistry-Srinivasan-Da...
4. Introduction to Food Engineering: https://www.sciencedirect.com/book/9780123985309/introductio...
5. Handbook of Food Engineering Practice: https://www.routledge.com/Handbook-of-Food-Engineering-Pract...
I was curious about how people with seizure disorders safely click links not knowing what lies ahead. Investigating the current best practices for building for epileptics, using this great resource (1) as my starting point.
This took me down a rabbit hole on current methods to detect seizure onset... I came across a very interesting journal article on applying ML in an implantable that can detect seizures within 3 seconds, which spurred my current research on less invasive detection methods. Like any good rabbit hole, I've strayed from the original mission.
Seizures seem scary and I don't want to give them to people, but the causes of their onset seem to be too nuanced and patient-specific to build with any guarantees. The best I can do is avoid the obvious and hope the cutting edge detection and mitigation research bears fruit.
1. https://developer.mozilla.org/en-US/docs/Web/Accessibility/S...
Figure skating. It should be illegal to have so much fun. The feeling of gliding out there on the ice with nearly zero friction is amazing. It's a great community too. Not cheap, but a lot cheaper than many hobbies. Less than $1k will get you a nice pair of skates, plenty of group lessons, and access to the rink for a few months. It's fun for the family or single adults. There are always tons of people just starting. Private lessons can get expensive, but for the average interested adult, you can probably do a single private lesson like once a month and then work on what you learned at the rink.
Spring Boot. What I learned is: I had no idea how much I hate "configuration".
Spring is a Java dependency injection system that uses XML-based configuration. Recognising that XML sucked, they later added 2 additional ways to specify configuration, which they call annotation-based configuration and Java-based configuration. Both kinds use annotations. Both kinds use Java.
Spring Boot is a layer on top of regular Spring that tries to make things simpler by automatically guessing what you're trying to do and configuring it for you, with something it calls auto-configuration.
Just trying to understand what makes what happen in a (very simple) Spring Boot app sucked weeks from my life.
Texas Hold'em. I know there are probably plenty of experts here but after decades avoiding it, I'm getting into it because I found a group of friends that played online.
I first discovered that there are tables (and tables and tables) of preflop card combinations that tell you whether to raise/call/fold based off of where you are in the dealing order and how many players are active.
Then I learn that's basically derived from monte carlo simulations, to calculate your chance of winning at any given moment (equity). It seems that it's probably more accurate to make your decision based off of equity and pot odds, i.e. Kelly Criterion. If I can get fast enough at estimating that in my head.
Also having fun trying to find open source libraries to do those simulations so I can create my own drilling exercises. Honestly, I'm having more fun doing that stuff than playing it.
The biggest dumbest problem is when blinds accelerate to the point that the minimum bet is greater than what Kelly would recommend. Some online poker sites are aggressive about this, to the point that you are forced to make irrational choices pretty quickly or get blinded out. A way to juice the house advantage I guess. It's almost enough to make me give up and find the next project.
Graph visualization.
I've been on a journey for a while to understand how to layout diagrams / graphs in an "aesthetically pleasing but structured" way. Long story short, DOT[0] is the best language I've found for defining graphs (compared to doing something with Mermaid.js or any other markup language), but rendering with the DOT engine in GraphViz fails the "aesthetic" test for me.
Did a bit of a literature review[1] to understand better the different approaches, and to understand the scope of the field. This book does great job of defining and providing the keywords for the different levels of requirements, starting with "principles" that are provable in the academic sense, to "conventions" that are like principles, but cannot be necessarily computed (eg NP hard, so requiring heuristics or simulations to achieve), and ending with actual "aesthetics" where things get very subjective.
Ultimately got pretty deep writing my own force-directed graph simulation in Rust and visualizing with egui[2] (needed an excuse to work on UIs and I've always wanted to write less Python), but I'm taking a break to use what I've learned writing Rust to shore up the REST API testing suite for my dayjob.
[0]: https://graphviz.org/doc/info/lang.html [1]: https://www.amazon.com/Graph-Drawing-Algorithms-Visualizatio... [2]: https://docs.rs/egui/latest/egui/
My two rabbit holes from the last ~year:
JellyFin. Amazing. A little clunky here and there, so some automation is needed. Then the weird bugs - gotta get a debug instance going to see what is going on. Once you start thinking about it as a "personal Netflix" you start building out a larger collection which needs organizing. Then your friends want an account. Then you realize your satellite tuner has a streaming function so you start reading through the plugin code for other streaming boxes...
Next one was the Japanese PS2 DVR combo unit. Sold as junk (which mostly they are - none can read discs anymore basically) but very interesting in that they are beautiful and cool and weird. The English OSD translations will often break your unit in subtle ways - so I joined the discord and identified that this was actually a configuration management problem - you have to only ever use a translation which was done to a set of OSD files which match your firmware revision. So I started writing a framework to auto-translate the XML file strings. Then the guys on the community mentioned the images with strings, as well as detecting when your translated string is too long for the UI element... I am still working on the framework. No dealbreakers yet! I will probably buy a few more to generate all my own OSD translations. My wife is going to kill me - I have 2 PSX DVR's now, and probably will buy more on our trip to Japan soon...
How the public transport services in my local area are run! There's a lot to it: route planning, scheduling, ticketing... and a large amount of digitalization too.
It's interesting to discover that a lot of 'facts' about public transport people take for granted just aren't true. The names and liveries by which vehicles go often don't actually correspond with their actual operators and owners. The company that's named on your ticket might not actually get any money from your purchase - or they might make money from passengers who don't pay at all, due to a myriad of subsidy schemes run by different levels of government.
Waves of privatization and re-nationalization with political motivations at every turn have produced a system which is amazingly efficient in some ways, and appallingly wasteful in others. Workforce strikes are obvious to the general public, but what's not obvious is who the negotiating parties even are, with various trade unions (and unions of unions) competing against various management groups (and groups of groups).
Some things are pleasantly surprising. Without any fanfare, digital systems for vehicle tracking have been introduced with remarkable efficiency. Then, for me, there's the astonishment of discovering that not only is every timetable published in a consistent, nation-wide data format, but one that has been utilised in production for twenty years!
It all makes me realise how limited the public discourse about public transport and 'green' mobility policies are in my region. It is simply impossible to grasp the true consequences of any given proposal in the meagre columns that they're given in the newspapers and the two-minute reports in which they feature on the television. Diving into this rabbit hole has led me to respect the complexity of the field much more than I did before, and fills me with both hope and despair on topics which I had hitherto scarcely lent a thought.
Bottle digging, i.e. looking for and collecting old bottles, typically buried or submerged.
Wife and I live on a floating house in Oregon outside of Portland. Taking my paddle board to various beaches on the river to pick up trash has become a beloved hobby of mine. Fantastic way to get exercise, get out in nature, and clean up my community.
I kept finding old, antique bottles during outings: Coke bottles from the 40s and 50s, little medicine bottles from who knows when, old beer bottles from brands no longer in business. I eventually learned that there's a whole hobby around this called "bottle digging", even an active subreddit (r/bottledigging).
I've since bought a nice snorkel and mask and have been diving for old bottles on clear days, when river visibility allows it. Thinking about taking it to the next level and getting scuba certified.
If you like finding things, you might like bottle digging.
Mine all feel like "we do these not because they're easy, but because we thought they would be easy".
Turning a thousand-page book - PAIP - into a stack of Markdown files in a git repo, readable online. The print book received more editing and revisions than the ebook. I converted an ebook's ... odd formatting ... into Markdown, remade diagrams, generated new ePub and pdf files, and had the spine cut off a print copy to make a fresh scan. Working on that scan, I made Scantailor, an X program, easier to access from a Mac, via Docker. I tried different OCR engines, and pored over the diffs, incorporating dozens (hundreds?) of improvements. I got to find so many differences between Markdown engines. I have ideas on how to make Pandoc links between chapters. There's still a lot to do!
My current WIP: Lars Wirzenius posted about file systems with a billion (empty) files. I started exploring because I was curious, if I was remembering correctly, how well a mostly empty image file would recompress - like, drive_image.gz.gz. Lars offered a Rust program; I was curious about how other methods compared. Like, how about nested shell loops, tar, and touch? And, hey, how well can we archive and compress them? I've gotten to see some issues, bottlenecks, and outright failure modes with SMR hard drives, Samba re: sparse files, and parallel gzip compression. I've accumulated some shell script boilerplate to make it easier to go back and verify my processes, and harder to accidentally wipe out past work if I rerun it.
I’ve spent a half months salary on literature auctions. French, latin, english, swedish, 1700s and up. History, memoires, philosophy, religion, science and math, poetry, stories. Art as well, happened to buy prints of Girodet and Dürer that I enjoy very much.
Did some duolingo for french just recently and already did for latin two years ago. It’s difficult to read them but broad context is enough for an overview understanding and I feel a detailed understanding increase by the chapter from exposure alone.
Apart from the actual books I enjoy bidding, similar to but healthier than gambling on stocks, and “winners curse” of overpaying for stuff I really like eh I’m aware of it but the dopamine is worth it too. Some arbitrage opportunities as well, as in finds on local auctions and selling online internationally, but usually international shipping for heavier stuff like entire œuvres and just the time investment to pack and ship something deters me from doing that. Wouldnt want to damage 1600s books in shipping I suppose as well, they look quite cheap locally but fetch more internationally.
This past month, I’ve been reevaluating my dev environment and workflow. My goals are to reduce RSI, be more efficient, as well as learn all my tools as deeply as possible. And have fun!
- I’ve ditched VSCode and gone all-in on NeoVim. I’ve spent a bunch of time watching Primeagen, etc., tweaking my vid config and learning how to navigate as efficiently as possible.
- Switched from QWERTY to Colemak-DH to hopefully reduce RSI. I’m at about 70wpm with decent accuracy after 4 weeks. My QWERTY skills are gone. I like Colemak, but we’ll see how I feel in another month or two.
- Finished my custom hot swappable Sofle keyboard, and spent many hours customizing the layout. I think I’m pretty close to feeling comfortable. I’m using home row mods, which I love. Currently using Kailh box whites (clicky). Might switch to Gateron Brown Pros.
- Been going through a “Build your own git” course, to understand git as deeply as possible.
Group theory.
Something is incredibly beautiful to me about classifying the kinds of symmetry things can have.
I’m trying to understand where the sporadic simple groups come from. Starting with the Matthieu groups. So far it seems to be due to some anomaly in Pascal’s triangle, but I’m still trying to put it together. “Another Roof” on YouTube has a good video about this.
I got into cross-compiling Python wheels (e.g., building macos wheels on linux and vice versa). Zig's `zig cc` does much of the heavy lifting, but one step in building a portable wheel is the "repair" process which vends native library dependencies into the wheel, necessitating binary patching (auditwheel does this for linux, delocate for macos).
I wanted to be able to do this cross platform, so I re-implemented ELF patching and Mach-O patching and adhoc signing in Python, and wrapped them into a tool called repairwheel: https://github.com/jvolkman/repairwheel
I've been doing some modernization on an old scripting language used by the game engine I work on [1]. Added a garbage collector, simplified how internal symbols are defined, added a VS code extension with some niceties like syntax highlighting, "Go to Definition", and doc tooltips. Also recently added support for websockets and plan to tackle JSON soon. Oh, and so much refactoring.
Our company decided that they want to make an electronic device. They've found an electronics engineer who devised a schematics and wrote some kind of PoC firmware. I wrote "driver" for it (really just wrapper around serial port) for our software to consume it.
That guy was busy with other tasks, so iterating on firmware was too slow. So I decided to dive in. I mean I knew C a bit.
So I had to learn STM32 arm, I had to learn low level C, I had to learn assembly, I had to get some understanding of those electronics things to get some sense of it, I had to read tons of manuals and datasheets.
Long story short, I rewrote this PoC firmware into something I could bear. It's so nice to control all software from the start to the end.
Now our company wants to rework this device into "smart", add display with touchscreen and stuff. So I'm digging into embedded Linux programming, LoL.
I'm generally consider myself full stack developer, so I can write frontend, backend, kubernetes, setup servers, deal with cloud stuff. However digging that deep feels like testing my limits.
USB! I’ve tried and failed couple times in understanding how USB works under the hood, from electrical, to protocol, then classes, and also Power Delivery. This time around things seem to make more sense now. It started out as an ambitious goal to emulate an FTDI USB DMX converter with the ESP32-S2/S3, but realizing that might be too big a goal, so I’m starting small. I want to be able to make a custom device class on the ESP32, and write a driver with libusb.
Metal-air batteries/fuel cells. Made a mini aluminum air battery (you can easily DIY one with household items). It seems that most people consider metal-air batteries to be a dead-end, since they aren't green and are generally non-rechargeable, and air cathodes are tricky (sluggish, exotic materials, expensive catalysts). I dove into "alternative" battery and fuel cell research after looking into how to extend the range of my electric motorcycle. I love the electric drivetrain, especially on motorcycles, but lithium ion isn't up to the task as far as capacity for anything beyond an hour or two of high performance fun. If I could get a compact metal air battery or hydrogen fuel cell to output just 1kw for a hybrid drivetrain, range issues could be solved.
I was writing an article on reverse-engineering vintage synthesisers[0], and I ended up getting majorly sidetracked trying to find out exactly what year the Hitachi HD44780 LCD controller was first manufactured, or try and find any background information on it. The earliest reference to it I can find online is a 'preliminary' user's manual[1], dated March 1981. I know it's a bit of a weird rabbit hole to go down, but I figured I'd get to the bottom of this mystery. It's a shame that there's so little background information available about one of the best-known ICs in history.
0: https://ajxs.me/blog/Introduction_to_Reverse-Engineering_Vin...
1: https://archive.org/details/Hitachi-DotMarixLiquidCrystalDis...
GoLang and Azure APIs.
I've been attempting to add a oauth2 device code flow to a Tacacs server with the goal of extending Azure accounts to access network device management planes. Pretty neat, I can get a "enter this device at URI" from the router/switch and let Azure do it's 2fa/compliance etc. Currently trying to get token validation working on the tacacs server =).
Ultimate goal is have a reverse proxy web front end kind of like Apache Guacamole that does the Oauth for the user and when they click on a network device, the JWT is passed through to the network device over SSH and thus the tacacs server which is relatively local to the network device which will validate it and let the user into the network device.
Playing around with GPT4/Opus a lot lately and man... I have feelings. They've been a great learning tool to learn the basics of Go though so I'm thankful.
It's going swimingly /s but I seem to be making progress. Slowly, I'll bake this into my bigger network management tool if it an be secure and make sense to do so...
Chrome extension build steps. Because the chrome manifest requires an actual file, you can’t rely on a dev server and hot module reloading. So I’ve hacked vites build command and watch command to constantly recompile chrome extension code as I develop it (typescript + frontend frameworks etc).
Add to the fact that you have to manually reload the unpacked chrome extension to apply your new changes, so I’ve hooked solution 1 up with a web socket server and a custom chrome extension that watches all your other extensions and talks to the web socket server to auto reload an unpacked extension anytime the build step completes
It’s a nightmare hack, but I may just be the worlds most productive chrome extension builder as a result! I’ve released 5 extensions in the last 5 months :)
About to release a big one, but it’s also probably the words most complicated chrome extension (LLM + Firebase store + Stripe + Auth + Serverless functions)
I've been bitten by the Meshtastic / LoRa bug. It's fascinating to see how far these little inexpensive units can reach.
This weekend I was able to reach my home node from a state park 8.2 km away and have been giddy since.
Back in the days my uncle told me about this new thing called Debian that he compiled over the last couple days. That was in the end of 1996. I went on to use it for a while in parallel to try things out, and was quite happy in the endless learning bubble.
Fast forward into my first job, there was a Windows machine with a crashed HDD and my task was to recover as much data as I could. Windows tools all sucked hard, even the ones that we had at the company that cost thousands of bucks per month. That's when the power of the Linux ecosystem hit me.
Went on to being an IT forensics guy, then went into pentesting, then into blueteaming and now I am having my own startup that builds a better EDR software.
I still have to think about that coincidence. Literally nerd-sniped my life, otherwise I probably would have still been a sysadmin or something.
On the way I learned a ton of new things, from programming languages and compiler bugs to exploit techniques and kernel development and even hardware design. If you go deep enough in hardware pentesting, the whole phreaking scene is amazingly welcoming. The CCC chapters are also amazing, and there's just so many opportunities to grow your knowledge and experience in the field. It never gets boring!
Chasing memory leaks in BCP [1] data load implementation in Babelfish [2]. Spotted unusually high RAM usage, decided to look deeper and got into a rabbit hole. Haven't had this fun of overnight debugging (of Postgres guts) for a long time. As a result found 4 different leaks and one (unlikely to be triggered) crash. And now have BCP data load impl with 100% constant-bounded RAM usage in a DB server process.
[1] https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view... [2] https://babelfishpg.org/
History of early Christianity. I found the gospel if Judas to be extremely fascinating.
Fun exercise to try and list them out! My last couple of weeks:
- 3D-printable parts storage solutions (via: I found some part storage bins in the discard pile at a local hackerspace)
- MITM proxy to snoop on Github Copilot API requests (via: we're building an jupyter AI assistant thing and got curious how other players do it).
- DIY robot arms (via: I'm making several for a nested 'you pass butter' joke, via a casual conversation about robotics being accessible now. YouTube is amazing at surfacing smaller makers once you start watching a few videos on a given topic)
- Learning about Oauth and JWT (via: 'why is auth still a pain?')
- Invertebrate UV fluorescence (via: that millipede is glowing under my UV torch!)
(a small subset of these end up documented https://johnowhitaker.dev/all.html eventually if you're curious to see a longer historical list)
I like rabbit holes where following the curiosity gradient to a satisfying conclusion is possible. "How does X work" leads eventually to code that does X. I'm less happy when they lead into a tangle of complexity, like digging into a library only to find weird abstractions 6 layers deep or trying to compare 18 different alternatives in a field I don't know very well.
OP I'd also like to hear yours!
Started like this. I had a few talks with startup founders interested in if they were hitting the market at the right time (their "why now"). Seeing more interest I ended up running a workshop on the timing topic for a few accelerators. I kept digging into what clues there are that a product will hit the market at the right time. Researched how the same products have been reintroduced again and again over years until finally they work out. Most surprising to me is how many people realize timing is important, yet how few have a framework to assess timing.
Fast forward a few years... I just wrote a book on the timing topic, called Why Now: How Good Timing Makes Great Products.
Ham radio!
I got into it through wanting some cheap radios to keep in my house, so I wouldn't have to go all the way upstairs when I needed to communicate with my daughter.
Well let me tell you Baofeng radios are extremely cheap but really flexible. I got these things for the simplest possible use case but after realizing their potential I just had to learn more about the space. You can adjust their configuration with a tool called Chirp and you're off to the races!
I attended a local severe weather awareness event where I met some hams who were part of an emergency response network. It's really cool to learn about how these communities operate. It's legal to receive even without a license - you only need the license to transmit.
I plan to take the technician test soon and get my license so I can help out at a nearby bike event. The area is incredibly rural so there's no cell coverage and the ham operators are really helpful in coordinating things.
Anyway, I feel like the hobby is a bit of a dying art, but it's something that seems like it would have a lot of appeal to the programmer crowd.
Golf simulation. I found myself in a unique situation where I had the space, computer and projector. So I got a mat, screen and the key piece, a launch monitor. Now I’m playing 9-36 holes of golf simulation almost every day. The tech is great, the setup is never ending. Golf simulation tech is blowing up right now too. I’m not even necessarily a golfer fwiw. But I love the simulator.
Rebuilding a Ford Cleveland V8 - the Australian specific 302 cubic inch version.
One broken engine and one non operational one and turning them into a single good motor. American thin-wall cast V8 engines are fairly similar, but different enough that if you don't get them built you have to do a bit of puzzle solving (especially in the timing case). Plenty of youtube videos and forum posts on the Cleveland and it's been fun piecing it back together and learning about new things like installing cam bearings.
Visiting and documenting abandoned mine sites across the U.S. desert southwest.
The big hole - I basically have learned the fun bits of STEM(3d Printing, Laser cutting, CNC, microcontrollers/electronics etc) over the last few years. I graduated with a ME degree 25 years ago but left engineering shortly after to go into the startup world. Built a career out of helping scale startups and now decided to retire and focus on my interest as I never had time or energy.
My current rabbit hole is tuning my home's boiler to be more efficient in its use of gas. It is an interesting engineering problem because the lack of feedback loop since the thermostat is dumb and specific home variables makes smart thermostats useless as well as boiler sizing more complex than most installers understand. My goal is to add some features specific to my home to reduce gas consumption based on more variables - outside temp, minimum outside temp, sun/clouds and home variables like brick wall temp, fireplace heat, boiler controller settings as well as the wifes 'I'm cold' variable.
I am using a microcontroller and custom current switch as well as IOTSTACK to send inlet/outlet temp and gas valve & circulator state (on/off) to influxdb/grafana so I can see what is happening between thermostat and boiler controller. I have identified a few freebies in terms of consumption and inefficiencies. I have added a relay to delay the gas valve once the boiler starts cycling to reduce "short cycling" which is a waste of gas on startup and a mini explosion every time gas lights. I have managed to reduce cycles in half which helps with wear and tear as well as the number of boom sounds coming from my boiler room :)
I would love to go down the simulink rabbit hole but I think I will not.
Trying to _really_ understand the postgres query planner's `EXPLAIN` output. We have long-running embarrassingly parallel processes where the throughput will sometimes completely tank. Got worse when we upgraded to PG16.
Trying to compare good query plans with bad ones, and then work out what changes we need to make to the slow queries is ... interesting.
I picked up a carbon-steel pan at IKEA because that was all that was left. I knew it takes some work to season. I did not know there's a whole community of carbon-steel nerds committed to getting the most perfect seasoning to fry an egg. Now I know more about the topic than practical or necessary.
I switched to Neovim from Sublime Text after trying copilot in Sublime, feeling sad, and then watching The Primeagen and his glorious mustache for too long.
Ostensibly I wanted to be able to code on the production server like a miscreant with the same tools as my laptop.
However I just wanted to regain command of my dev environment after years not coding.
I also reorganized the furniture in my office and got weirder lighting to make it hacker friendly. I bought a new desk to solder electronics.
Most people know me as a partnerships marketer or product manager but I am a compsci at heart. This made me happy.
"Recently" (~5 years ago) I got very interested in mathematics, and decided to self-study up to an undergrad degree-equivalent level.
I researched what topics are typically included in a 4-year math major university program and what textbooks are used to teach those topics at MIT. Then started grinding all the way through from beginning to end.
It was so awesome that upon finishing, I promptly started all over again... but with physics instead.
MH370. There was no intermediate step as I saw somewhere it was the 10th anniversary of its disappearance and decide to get a quick update on the current status of all the evidence and theories. Ended up spending the entire day reading about radars, pings and aviation controls.
The wonderful world of research paper identifiers.
DOI (Digital Object Identifiers) are used by many modern research papers as sort of a UUID for papers, run by doi.org.
But they're discipline-specific. So they're used widely by certain disciplines. But others use different databases.
So for biology-related papers, NIH's PubMed ID. Or for Astronomers, Bibcode.
All are "global" identifiers and each has some kind of consortium that's trying to make theirs the One ID. DOI seems to be the closest.
Man, I envy the dedication and energy y'all have. I retired 10 years ago and have completely lost motivation and the ability to stick with anything.
Ooh, another one I have: I cosplay as a sysadmin.
A while back I had bought a domain for my email, and I thought “I should write a blog about creating a blog”. At first I hosted it in GitHub Pages, but then I realized I have a perfectly good Raspberry Pi. It’s not like I’m ever gonna get a lot of traffic… so why not self-host?
That sent me into a very deep rabbit hole. How do I make sure my website doesn’t go down if my IP address changes (no static IPs for me, sadly)? How do I create and automatically renew a certificate? How do I achieve high availability?
A few years passed, and now I have a cluster of a few Raspberry Pis running Docker Swarm, managed by Portainer, with stacks running multiple websites and services I self-host. I’ve learned a lot!
My next move is going to be a full overhaul: Docker Swarm is blocking me from setting up some things the way I want to, so I want to build a new cluster using Kubernetes. I’ll use the opportunity to overhaul the network layout as well.
The funniest part is that I haven’t written a single blog post in 3 years. I wanted to add responsive images so I could add diagrams and photos. Somewhere along the way I realized I shaved too many yaks.
Why doesn't my Powershell terminal work in Visual Studio Code?
What do you mean my launch.json file is missing? It was there yesterday?
Wait, I can set up custom launch settings in my launch.json? What else?
Ok, so I've got seven different launch settings in there, and now to see if I can have one used for markdown for my markdown word editor.
Oh, neat, lots of extensions for markdown.
Wait, you can install vim?
An hour later, and I've completely re-broken my VSC and am reinstalling from scratch.
Was into cine drones for a while; then discovered FPV. Spent dozens of hours on simulators, now flying actual FPV drones in manual mode. Incredibly addictive. I'm having a hard time focusing on anything else right now.
Suno and Udio.
Spent WAY too much time adapting a Buddhist sutra into a heavy metal banger: https://www.youtube.com/watch?v=H-5Y9Z7DK4s
I got a ZSA Voyager split keyboard and then spent weeks exploring custom layouts. The first question was QWERTY vs something better. Then there was layers and layer navigation. And should I swap out the key switches? And Keyboard Maestro.
Now I'm trying to abandon 30 years of muscle memory and typing at 4 wpm while I learn Colemak-DH. Maybe what I should really do is build a custom 34-key board...
I've been researching and planting fruit trees and edible plants. Looking at paw paws, peaches, pears, berries, persimmons and tea bushes.
Reverse engineering android apps. I wrote a bit about it in [0]. In the weekend I also started doing another one. It's interesting to see how these apps behave.
[0] https://github.com/benhamad/blog/blob/main/2024-04-12-dramal...
I couldn't find any containers for paper filters for the AeroPress XL that didn't look like cheap 3d-printed garbage, so I've been going down a rabbit hole of how to build bronze articulated joints.
I'm building a paperweight inspired by vintage brass table lamps to hold the papers in place on a wooden platform.
Got a ARTinoise Re.corder (an electronic wind instrument, basically shaped like a recorder, but MIDI) for my wife. Works with an iOS app for basic sounds. She's spent very little time with it :-)
The REAL rabbit hole is the astounding amount, and quality, of AUv3 plugins for iOS. Sounds, effects, looping tools, MIDI things, just... wow. And almost all of them are under $20, and many are free! I've spent less on a dozen software toys than on the first two guitar pedals I got. And infinitely more powerful.
Check out this video of someone doing the looping thing way way way better than I'll ever be able to (but it's fun to work towards a goal). Software she's using is called Loopy Pro, another amazing thing:
I spent the last week evaluating drag and drop form designers and came to the reluctant conclusion that I’d be better off building my own. Long story. So now I’m building a drag and drop form designer, quite the deviation from the road I need to travel down.
Its almost certainly a rabbit hole but at least I’ve forewarned myself.
Lots of really cool answers here.
Electric Bikes: Hub Motors and Mid drives are a really great spring / summer rabbit hole to go down. So many form factors of ride and you can also kill two birds with one stone by going for 60V lawn care equipment (There's adapters on eBay to connect them to your bike).
Two very very deep rabbit holes in the last 6 months:
- Designed/built a small USB controlled pan/tilt camera head to control the mirrorless I use as a webcam (couple of servos, gears, belts), and then designed/built a custom ortholinear keyboard with a joystick to control the camera (custom PCB, CNC'd aluminum case, etc)
- I'm a pretty big runner, built my own web based calendar UI that integrates with Google Calendar where I can type in workouts like "1 mile warmup @z2 + 5x(30 seconds @ 6:00/mile + 0.5 miles recovery) + 1 mile cooldown" and this gets parsed/total weekly mileage gets tallied. The next step down this rabbit hole is building a small iOS app to automatically generate Apple Watch Workouts using WorkoutKit.
Well... I wanted a dedicated AI/ML box here at home, with enough horsepower to do some interesting AI stuff (but without breaking the bank totally). So I built myself a new PC from parts for the first time in around 20 years. That was ... interesting. Building a PC is - in many ways - not that different than it was in 2005. But it's also changed in many ways. So I spent a lot of time researching and picking out parts, and then building the system. Which led me to things like spending hours watching youtube videos on "What's the best thermal paste" and "what's the best CPU cooler" and "are thermal pads better or worse than thermal paste" and "Do you need one of these CPU gasket things if you're running socket AM5" and so on.
I just got the box finished up over the weekend and it's working really nicely so far. I went with an MSI MEG x670e "Godlike" motherboard, AMD Ryzen 9 7950X CPU, MSI Gaming Radeon RX 7900 XTX GPU, and 64GB of DDR5 RAM. The whole thing is running Ubuntu 22.04 (I was going to use Alma Linux, but the Alma installer wouldn't even start), ROCm 6.1.0 and Ollama. So far I've mostly been working with the LLM stuff using Ollama and the llama3 model.
Now I've started using Spring AI to interface with the Ollama API. Next steps: figure out "function calling" with Ollama (which doesn't seem to be supported by Spring AI yet, boo) and some "agentic workflows" and multi-agent stuff...
Apologies for the metacommentary but do you understand what you've done here?? You've collected together a bunch of rabbit holes into one all-powerful thread that some people might never be able to leave! So I'm gonna keep it bookmarked for later, thanks...
I haven’t had time to even look but etcd is on my list. It is a distributed key/value for node configuration (I think!)
The reason why I want to learn more about it is I feel it is like a base building block of distributed systems and may be easier to grok and even write a toy version than a bigger thing like kubernetes or a leaderless distributed datastore. I would also learn some go and know how a critical piece of kubernetes works.
What led me there is practicing for a damn system design interview. As much as this whole topic is controversial on HN the grinding has really got me curious about the tech that runs at larger scales and how it works under the hood.
Unrelated to work: the black hole rabbit hole.
It started with an article about the hypothesis that planet nine may be a primordial black hole with 3-6 Earth masses.
What’s a primordial black hole? It’s one that formed in the first seconds after the Big Bang. We don’t know for sure they exist but many theories and simulations predict them.
They’re an excellent dark matter candidate. Could it be that simple? Could at least a lot of the missing mass be tied up in little baseball sized embers from the birth of the universe that rarely interact with anything so we don’t see them? They’d be small, would rarely interact, and unless they are sucking in mass (causing a hot accretion disk) would be dark.
Then I got onto Hawking radiation and whether micro black holes could exist. Along the way I read about loop quantum gravity (LQG) which looks to me like a decent stab at unifying QM and GR that’s much less baroque and more testable than string theory.
That then led to the LQG “bounce” hypothesis for black holes. See LQG does away with true infinite mass singularities. Instead a black hole would be matter packed to its theoretical maximum density (which is still insane). From there it would quickly “bounce” and become a white hole.
So wait… how do black holes persist then? Time dilation! From a the hole’s frame of reference it collapses and then instantly bounces and goes kaboom. From our frame of reference though all that gravity slows it to such a crawl that the black hole phase at or near max density looks like it’s stable. The bounce takes billions to even trillions of years!
Last but not least I learned about the black hole starship idea. It’s a set of ideas about how far future intelligences could use black holes as mass energy converters to reach relativistic velocities. Might be somewhat easier (for crazy sci-fi values of “easy”) than handling antimatter. This also gives SETI yet another wild extreme technosignature to look for.
… and back to the beginning I found a post about how if planet nine were a PBH we could use it to yeet probes to the stars at meaningful fractions of c… at least if we could make them able to survive insane g forces. Unlike the black hole starship this would be feasible today. It’d just be a gravity assist off a ludicrous gravity well.
Here I thought black holes were dull. Turns out they’re the most extreme objects in the universe and a whole lot of the most amazing physics intersects around them. If there is any way we could tap into the phenomenon we could potentially access sci-fi levels of energy too.
Building a CRUD app.
It's been a while since I've worked on a CRUD app so I'm finding the whole thing quite interesting. The purpose of the app is to solve a scheduling problem.
I've written my own CDCL SAT solver (now just using google or tools), and on the app side I've jumped from Phoenix (elixir) -> Dream (ocaml) -> axum (rust) -> Django. I feel like Phoenix probably perfectly suits what I'd like to do with this app (long running tasks and collaborative editing) but I'm at the point where I want to support this app long term and I don't see me not being familiar with python anytime soon.
"What if I were to gather these 5 five recipes that really worked in a future... book?."
Bookbinding has fascinating details.
Due to all the recent BSD posts I have spent the past month exploring NetBSD and OpenBSD. Really enjoying the journey and finding that I can do everything I am currently doing easily on both.
I tried creative coding. Sonic Pi, Processing, p5js, there are so many frameworks built around creative coding that I didnt know even existed. It was really fun.
I'm slowly simplifying all of my projects. I'm constantly aiming for simpler, lighter, lower maintenance. Nginx for caddy, PostgreSQL for SQLite, shorter configs, static site generators.
It's fun to reach a point where things just run forever at zero expense.
In parallel, I work hard on developer experience just for myself. I finally get the greybears and their keyboard incantations. No UI can beat scripts and muscle reflex.
Installing and exploring V7 unix (1979) on a PDP11 emulator.
Crazy how familiar and yet different things are.
ERC-404 draft standard
an amusing AI generated origin that entertains the humans sufficiently well, while solving some frictions the humans encountered
basically somebody launched a crypto token and rugged it for some quick cash. it was a hit because their code, coaxed on and perhaps entirely generated by ChatGPT, did fun things:
- issues an NFT from a collection whenever you buy a whole token, but not when buying fractions
- transferring the NFT transfers the whole token
- while selling a non-whole unit of the token irreparable burns the NFT
community said “no that was awesome lets keep doing that” but original dev could only think about doing more rugs, so some in the community made Pandora token as the new reference implementation and pushing this draft standard’s development along
in reality it solves some frictions with token + Nft launches, any NFT collection launch, and multiseries management within 1 contract. Something ERC1155 tried to achieve in a more complex way. now people can launch NFT collections directly from an AMM’s liquidity pool like Uniswap. No direct boutique websites or NFT marketplaces needed to trade the NFT or create liquidity. And has a fun incentive for holding on to the tokens.
so far in practice, communities make separate markets for the NFT's with them trading at a premium to the token, as the NFTs in the limited size collection gets burned over time by sellers and has its own scarcity attributes while the fungible token trades at a different lower price, allowing for roundtrip arbitrage. Automatic community with the arbitrage features. ie. a Pandora NFT trades at $7,000 and comes with 1 token. While the token market alone trades at $5,800.
AWS Bedrock. Set it up to experiment with different models in one place. Turned to be fascinating; UI is far more friendly than AWS usually is. Cool stuff:
Guardrails: No longer do we have to use the AI to fight AI, AWS has presets for blocking inappropriate content going into or out of the AI block.
Evaluation: Basically just give it a JSONL file of all the inputs and expected outputs, and compare two models or judge the quality of a model. This can be done by a human (it even manages the access of this team of humans). But more interestingly, it can be automated or done by AWS workers.
Knowledge base and agents: Seems like it's only for Claude, but damn, Claude has been impressive for a good price. Claude Sonnet (the mid-range) has been about as good as GPT-4 for half the price. The weakness has been things like knowledge base and AWS lets us just quickly set up vector DBs and embeddings. Then the agents feature lets us connect with it and combine it with all of the above.
Bluetooth coffee mugs. I disliked Ember’s app, particularly on Mac, so I built a substitute for the menu bar. While expanding to knockoff mugs on Amazon, it was entertaining to see how bizarre and inefficient some BLE implementations were. Some write every second unnecessarily. Ember starts the heater during cooldown, even though they could do so closer to target temp and save 5-10% battery.
Writing a low level C code editor, going down the rabbit hole of gap buffers and piece tables. Fun though.
I first wrote it the dumbest way possible, one big array with padding at the back. Worked fine actually for most modern use cases, but as this is also a learning experience, I want it to be best in class in performance.
I think I'll settle on gap buffer because the performance is great and it doesn't hurt my head.
A quick and dirty, shallow one, that I just opted to brute force out of curiosity.
An online game I play includes an optional two player Russian Roulette type feature (non-fatal). I got to wondering if there was an optimal betting percentage to use, if you set aside some money as a betting seed. So I spent time coding up a really ugly brute force "just run lots of games and see".
Pretty much the answer is you'll lose more often than you win, looks like your best bets are around 2% of whatever money you have left of your betting money.
If you play 75 games, at 2% of your betting pool, you'll come out ahead only about 49.8% of the time.
There's more efficient ways of working that out than I bothered to do, which was to create a basic abstraction for a gun. For example, your odds of winning is essentially 50%, given two players. For every "game" I simulated, I could have just picked a random integer between 0 and 1 instead. Faster and the same effect.
As best as I could find, there are no good betting strategies on a coin toss (which is what this really is)
Formal Methods; and i don't think i am going to get out of this rabbit hole anytime soon in the future.
More than just using some formal language/tool i had wanted to learn about the Mathematics/Ideas behind formal methods and how they are embodied in TLA/Z language/B method/etc. After a survey of available books i zeroed in on Understanding Formal Methods by Jean François Monin hoping to get an overall idea of how everything comes together. But what i got was a fire-hose/mishmash of so many different sub-fields/notations/abstractions used in the field that it is quite a struggle to get a good grasp on anything. The author's style of writing is obtuse/challenging and the contents are more of a survey/introduction than detailed explanations.
The result is that i am now interested in figuring out a whole lot of mathematical/logical sub-fields which i suspect is going to occupy a lot of my time in the future.
After the post the last week about the DIY GPS receiver, I decided to get out my RTL-SDR and set it up. Took a couple of days of fiddling around with it, but now I've got SDRtrucking setup. Spent one morning listening to the public safety radio traffic. That was a wild ride on it's own with all sorts of things going on in this metropolitan area.
So, I was looking at some performance issues which seemed to be stemming from linux networking stack. We eBPF with docker/k8s a fair bit as a PaaS, and I ended up getting into the weeds with linux sys/procfs and kernel tracing. One thing led to another and after quickly dispatching the perf issue, I was supposed to be on a planned week long holiday which turned into a deep dive into linux kernel + networking.
Intermediate step - I feel pretty confident with the gory details of the kernel code now. can possibly build a custom kernel, boot qemu with both a simple C+assembly bare metal kernel, or the self compiled kernel. I feels like the clouds have cleared and I can see the sun. - Incidentally the kernel source code is pretty well documented, but one thing which is missing is a much smaller list of files which are most important. true pareto here. 20% files carry the weight. You also need to know the subsystem you want to touch. Chances are that subsystem is much lesser number of files.
Finally - Got to reading about kernel packet handling. at the L2/L3/L7 level. from nic hardware to userspace. Turns out that eBPF [hello old friend!] has a networking avataar called XDP which is pretty recent [<5 years] way of doing high performance networking on the linux kernel. Along the way, got to know about network performance optimizations specially in modern multicore systems in the kernel like RPS/RSS/aRFS, DPDK/fd.io/VPP.
And now I feel the itch to apply this to some of our networks. Particularly, baremetal servers on equinix metal + aws ec2 + azure can be peered with either VPP/Bird to make a p2p connection which is a factor more performant than the vpc interconnect/gateways which are provided off the shelf.
I might extend the holiday by a few days. and I would love to talk to people who have hands on experience with any of this. Its hard to contain my excitement tbh.
Video console emulators and AI. Emulators made me dig into video games history, which some I know first hand. I'm utterly amazed by the tools people is developing everywhere. AI, well it marvels me every single day. Funny enough both topics end up hitting again and again another obsession of mine, the dreadful laws of copyright.
Writing a beginner intro to wealth tax and assemblng and organizing all the Ressources I had consumed in years. https://start.me/p/1997pL/inequality-for-the-pragmatic-ideal...
How to improve one's speaking voice. Mine is tiny, no strength, no quality, monotone. Found something called "kharaj ka riyaz" and practiced it for a few weeks and voice quality and strength improved. Then quality went down and my voice is not irritable and scratchy. I hope it recovers.
I didn't start out planning to write a text layout engine. But that's the way it ended up. I need a different hobby!
So many great rabbit holes in this list. There are several that have blown years of my spare time.
My one, at the moment is precision time keeping. Time nuts. I have a pile of oven controlled crystal oscillators, GPSs (even the full Ublox time specific LEA M8).
I got a BG7TBL counter and multiple cheap GPSDOs to test my own.
I have a DAC1220 20 bit DAC on an ESP32 disciplining a TCXO from an old phone base station by counting the 10Mhz using the PCNT and gated off the GPS 1PPS.
Meantime, I learnt the esp-idf so I could have more control over things. Everything done with the esp-idf is way way more stable than using the Arduino wrapper, no idea why, maybe later versions?
The disciplining/tracking parameters are exposed by http and mqtt and put into influx.
I have a 5.5 digit multiple meter (I repaired the classic HP 3478A). Maybe I need another digit, there goes another rabbit hole. Voltnuts.
Juggling, I found a link to the library of juggling in HN [1]. I've been practicing for the last month and learning basic 3 balls paterns.
Been tinkering with my personal site again for no particular reason.
1. Looked at what it would take to turn it into a sort of "pubnix" for some friends
2. ...which got me looking into how to set up Postfix to manage local emails (allegedly this works out of the box, but I must have screwed something up since I never did get my test messages from one user to the other)
3. Then on to looking at BBS systems, starting from Enigma 1/2. Didn't get too far into that since the theme customization scared me away (and not enough of my RL friends are nerdy enough to get into it)
4. Finally backing away from the pubnix thing again because of insufficiently nerdy friends (although one is humoring me in experimenting with SSB), I then instead set up a Synapse server to have my own identity in the Matrix ecosystem.
Oof, Pulumi here. A module was wrapping a Terraform module which wraps a REST client which wraps the provider’s API. Errors only came forward during a `pulumi up` instead of being caught during a `preview`. Fun times digging through those multiple layers of abstractions
Bartending / cocktails / "mixology" .
It's more fun than I thought because whipping up every cocktail you've ever heard of is actually quite simple. There's also some cool generalizations to uncover so that you can Pareto principle your way to knowing how to do almost everything with a few tricks and ~10 liquids. I could have someone knowing more than 2/3 of bartenders (most are just a job / basics) in an evening. Then once you've got a good handle on all the basics, there's an endless adventure into variations and history. There are a lot of flavors out there you wouldn't believe existed; many concoctions of herbs and botanicals. It's a great social activity too.
I installed NixOS and went down into that black hole.
I take photos for fun in my spare time and recently started shooting on film again. This time around I've been interested in how film actually functions. SmarterEveryDay has a fantastic 3-part YouTube series of the Kodak manufacturing process as he tours through the facility. I'm now amazed that film is still even being manufactured today, and can't even imagine what the production line was like during the heyday of film.
I've developed my own film in the past but knowing so little about chemistry myself, it's still pretty much magic to me even after digesting all of the info from the series.
Miniature painting
I wanted to be able to easily distinguish between the 60+ figurines of the Zombicide board game, so I figured I could paint them "quick and dirty".
Well 8 months later, I'm not finished because I "had to" learn about paint, color theory, paint mixing, human vision, brush types...
Being colorblind I gave no attention to colors around me, but I have since discovered I can see more shades than I was aware, and I'm having a blast just looking at the foliage... Which does not speed up the painting!
I have spent the. last few years (especially during COVID) to collect and explore the world of stationery. I now collect over 2500 items ranging from pens, pencils, stickers, a 50 yr old typewriter (functioning), note pads, old diaries, planners, gifting items, collectibles like from army, navy sets of pen, card holder and so much more, from around the world. It fascinates me to know such an amazing rabbit hole exists.
Everything. Everything I look at or touch I soon realise it’s way bigger and deeper than I thought. You could choose nearly any topic and turn the study of it into a phd if you really want.
I'll try to build my own corne keyboard so I recently dived into custom mechanical keyboard rabbit hole, which seems to be more of a black hole from which nothing comes out, not even light. (Plot twist: I have no experience in soldering, so that's gonna be fun...)
Otherwise, ThePrimagen pisses me off always talking about Neovim and Vim motions, so I am right now in the painstaking process of learning Vim motions and I want my life to end because of the learning curve.
This is pretty fitting for HN, especially considering I've tried to ignore it this whole time... but AI.
Watched a couple videos on making AI to play video games, looked into localai and gptpilot, failed to get it to write a QT application. Played around with tts and stt, and now I'm stuck in prompt hell with diffusers/dreamshaper.
(It's mostly there, but I feel like it's screwing with me because it always finds a way to screw up the picture.)
I've been interested in making a homebrew playstation 3 (childhood console :D) game for a while, and finally got an environment setup yesterday.
Ended up getting a simple rust function built, with some slight miscomplilations via wasm + wasm2c. Now I'm going to try to get graphics working.
There is a surprising amount of public code containing calls to sony's licensed SDK, if you know what to search for (Not to mention the SDK, which was, "obtained" dubiously). Fascinating stuff
I have bought recently a bunch of ICs from 8080/8086 computers (mostly peripherals like PIC, I/O, Timers, memory) for next to nothing and i am trying to build a "retro" synth. It is way more fun and challenging than arduino or esp32, and not really complicated! A lot of instruments of the (Roland Juno, Oberheim Matrix 6, Akai AX-80) are using these chips too, so when in doubt i can take a peek at the schematics and move on with construction.
The Rust 3D rendering stacks. I just want to use them, not work on them. But there are low-level problems and I had to open the lid on Rend3->WGPU->Vulkan.
Over the last little while, I've fallen down the Nix rabbit hole.
I don't currently use it in any serious projects aside from tinkering about with it, but it has been a lot of fun to learn and study.
Between the Nix package manager, the associated language, etc., there has been a lot to learn about, and it's been good fun. I have nixOS on my spare Thinkpad for toying with, and I have Nix on my main Debian systems, if I want to pull something from nixpkgs.
I started looking into the Gospel of Thomas right before the start of the pandemic, which led to a number of rabbit holes:
* Turns out the work is not 'weird' or 'Gnostic' but is directly addressing details from Lucretius, including paraphrasing his view of evolution and atomism, but refuting the claim there's no afterlife by basically appealing to the idea we're in a simulated copy of an original physical world where the spirit doesn't actually depend on a body, because there is no actual body.
* As I dug more into the various mystery religions the followers of the work claimed as informing their views, I saw a number of those were associated with figures various Greek historians were saying came from the same Exodus from Egypt as Moses.
* Turns out a lot of the ahistorical details in the Biblical Exodus narrative better fit the joint sea peoples and Libyan resistance who end up forcibly resettled into the Southern Levant latter on. In the past decade we've also started finding early Iron Age evidence of Aegean and Anatolian settlement and trade previously unknown in the area, including in supposed Israelite settlements like Tel Dan, lending support to the theory that Dan were the Denyen sea peoples.
* Also turns out that in just the past few years a number of Ashkenazi users have been puzzled by their genomic similarity to ancient DNA samples, where the closest overall match in a DNA bank was 3,500 year old Minoan graves sequenced in 2017 or that they have such a high amount of Neolithic Anatolian (which the 2017 study found was effectively identical to Minoan).
* The G2019S LRRK2 mutation that's almost only found among the Libyan Berbers and the Ashkenazi appears to have originated with the former but appeared in the ancestry of the latter ~4,500 plus/minus 1k years. Which is a window that predates the emergence of the Israelites in the first place, but is on the cusp of the sea peoples/Libyan alliance.
* There's also been discovery of endogamy among some of the Minoan populations. Did the Ashkenazi endogamy evidenced from their emergence in Europe and the bottleneck in the first millennium CE actually go back much further than we've been thinking? Maybe Tacitus wasn't so off base when he talked about how some claimed the Exodus involved people from Crete hiding out in Libya.
Anyways, that's a very rough summary of some of the rabbit holes I was going down.
Bonus: Herodotus's description of Helen of Troy spending the whole time in Egypt has two datable markers to the 18th dynasty, which is when Nefertiti, "beautiful woman who arrived" is around during a complete change to Egyptian art and religion while she's the only woman in history to be depicted in the smiting pose, with her only noted relatives being a sister and wetnurse.
My project uses a library, which in turn uses a WSGI library. During a vulnerability scan, my web UI would crash, but I had no error messages. The rabbit hole!
Despite leaving a bug ticket with my library, I received no response after a month. I struggled to debug this library within a library. One day I realized the rabbit hole I was in and switched over to Tornado and no problems since.
I’m learning LLM and AI. And I’m building a multi-modal full stack LLM chat agent. And I got very fascinated by it. [0]
Using semantic-router for dynamic conversation routing, and LiteLMM for model providers.
It was lots of fun to learn and build. I will be adding function calling support (tools use) for the models to have it more capable, like an agent, in the future.
I watched Lex Fridman interview Richard Wolff and have spent 2 weeks going hard into marxist and anarchist theories and practice. Working through 2 books, a dozen browser tabs, interviews, etc. It's rare something catches my interest like this (especially non-technical). But I'm really enjoying all the different perspectives and formulating my own fantasy scenarios.
Used tensorflow since its creation until changed companies one year ago where I only use torch lightning. I'm fixing lighting to make it have some of the nicities that tenorflow has. Also finally fixing the graph engine in tensorflow so the abstraction actually works and doesn't have sharp edges around things like models of models.... aka graphs or graphs.
Several renowned mechanics in my area don't work on cars that are older than OBD-2. So, I went down the rabbit hole for:
One person brake fluid replacement / bleeding procedure after replacing a brake master cylinder. For my car, I can now do this without taking the wheels off. I'll be done within a couple of days and after that, I'll be kind of an expert.
Collatz conjecture
I found my two old mechanical PC keyboards from the mid 80's in the garage, took them in and cleaned them. Bought some VIAL adaptors for them and new alps keycaps for one of them. Ended up reading about mechanical keyboard history, keycaps, VIAL/VIA/QMK, etc. Also bought an old IBM model m keyboard yesterday.
Been nerdsnipped and diving down the rabbit hole on a few topics in the past few months:
Some history podcasts had me digging into the Napoleonic Wars and Israel/Palestine.
Also a recent interest in human health and diseases has basically sent me down the path of self-study equivalent to a Kinesiology/Exercise Science/Sports Physiology degree.
OpenWRT, CD and DVD Players, Kindles, and OpenBSD due to some recent HN article.
I knew about most of them a little or fair bit already, but there's always something to learn the deeper you go :)
In browser Time travelling debugging in Javascript.. trying to implement it for an open source hobby project
I’ve been trying to fit something into GCP free tier while also designing it to be scalable.
Been tinkering with that for an absurdly long time vs just throwing some python onto a $5 dollar VPS but it’s been fun & learned a decent bit about over engineering pitfalls along the way
The temple os , terry a. Davis one
How to break substitution ciphers.
Somehow and unintentionally as the search began from some random article I’d read, this seemingly unrelated subject ended up uncovering some insights into a problem with deduping database rows I’d been working on for another project.
I'm trying to beat level 19 in terminator dash: https://www.atarimania.com/pgesoft.awp?version=37332
I've been fascinated with the VESC project (https://vesc-project.com). I had no idea that controlling a BLDC could be so complicated.
I used Termux to get Simh running in my android based smartphone.
I then set up a VAX 11/780 running OpenVMS 7.3, which I can telnet into from outside. ;-)
Local brewery is doing a cinco de mayo event and we started talking about pre-gaming with margaritas:
Margaritas -> Jello Shots -> Chimoy/Tajin rim/topper -> Pop Rocks -> History of Pop Rocks
Jpeg-xl encoding. Its API is almost openssl-style confusing. I found no way yet how to encode a image buffer to a file.
ChatGPT gave some nice code, but it wouldn't work at all, good ideas though
Started on the Nagoya Protocol, then PCR of wastewater on airplanes, mechanical engineering of lavatory fittings, then metagenomic shotgun sequencing, and now Bloom filters.
I'm playing Zelda Tears of The Kingdon... almost infinite side quests..
What is the difference between sony camera models.
Today in Toronto, Canada, a man that was charged with First Degree murder of a police officer was acquitted. I'd only really absorbed the initial news story in passing. The initial news releases relayed the police account of the incident (a man running over a police officer with his car) that it was a "deliberate and intentional act". During the trial, and with today's acquittal and subsequent lifting of a publication ban, we learned how different the public perception of the initial events were compared to what actually transpired. Thus, it's interesting to use technological tools (search, or wayback machine) to see how the story was originally framed and how subtle differences can affect perception. My hope is that the event at least makes people question their own assumptions borne of limited or misleading information in the earliest publications of any news event involving the death of a police officer.
Even today, I'm seeing conflicting reporting of quotes. For example the CBC has one article that quotes the police chief:
> Outside court, however, Toronto Police Chief Myron Demkiw struck a different tone. "While we respect the judicial process and appreciate the work of everyone involved in this difficult case, we were hoping for a different outcome," he said.
https://www.cbc.ca/news/canada/toronto/umar-zameer-acquittal....
However, the video of the Chief's statement is slightly different (and quoted correctly in another article):
> "While we respect the judicial process and appreciate the work of the 12 citizens who sat on a very difficult case, I share the feelings of our members who were hoping for a different outcome," Demkiw said.
https://www.cbc.ca/news/canada/toronto/umar-zameer-verdict-1...
The first quote cuts out essential information and does not note that the quote was not verbatim (as presented). It's especially problematic as the quotes might have subtle but different interpretations now that his suggestion of "hoping for a different outcome" is being addressed. (There's a subtle difference between 'sharing feelings with someone that hopes for a certain outcome' and 'hoping for a different outcome' -- and this could be major if it's seen as the police services suggesting the wanted the man, who was declared innocent, to have been convicted.)
(One other possibility is that the Chief did in fact mention both lines, and that they are so similar because it was based on his official public statement).
San Francisco politics in the late 19th century.
I recently fell into internet comics from the early 2000s. Stuff like Absurd Notions, Sluggy Freelance, and Freefall
----esp32 module ----gaussian splatting ------
People keep talking about blimps, but they neglect the price of helium which is rising.
A while back I wrote a game in assembly, for CP/M. Since I have a single-board Z80-based computer on which I can run it.
I later ported the game to the ZX Spectrum, because that was a fun challenge, and I only needed a few basic I/O operations - "write to screen", "read a line of input", etc, etc.
It occurred to me that I could reimplement the very few CP/M BIOS functions and combine those implementatiosn with a Z80 emulator to run it "natively". So I did that, then I wondered what it would take to run Zork and other games.
Slowly I've been reimplementing the necessary CP/M BDOS functions so that I can run more and more applications. I'm not going to go crazy, anything with sectors/disks is out of scope, but adding the file-based I/O functions takes me pretty far.
At the moment I've got an annoying bug where the Aztec C-compiler doesn't quite work under my emulator and I'm trying to track it down. The C-compiler produces an assembly file which is 100% identical to that produced on my real hardware, but for some reason the assembler output from compiling that file is broken - I suspect I've got something wrong with my file-based I/O, but I've not yet resolved the problem.
TLDR; writing a CP/M emulator in golang, and getting more and more software running on it - https://github.com/skx/cpmulator
I was Youtubing this last weekend and ran down quite the rabbit hole. This all seems like mumbojumbo to me.
I could not derive a single piece of solid science in any of it.
It was remarkable how much content there was on this subject with little to no actual information - enjoy:
Sound event recognition is so interesting and seems to be lagging the other ML domains
Color theory, human color vision, color models and so on
Tracing the lineage of the British royal family back to Elizabeth I.
The Night The Stars Fell
I created a startup 2 years ago around a gigantic rabbit hole: understanding cause-and-effect and how you can move towards the outcome you want in a consistent way.
In case you want to have unlimited fun yourself, ask yourself: "What is the purpose of X?" and then "how can you measure/assess the fit-for-purpose of it?"
Possible Side-effects: #1 you might get disgusted and even angry with the self-declared "experts" who have not even understood the basic concepts.
#2 you might learn how little you understand yourself and how deep the rabbit hole goles.
Example in Software Development: Understand the quality dimensions for a "definition of ready" and what impact a good/used DoR has compared to a bad/not used DoR for the efficiency and effectiveness of a software development process.
cheap chinese retro consoles. Bought one december, now i have seven with different linux configurations. Also Steam deck.
Saw all the CSPAN videos for the hearings on the January 6 2021 attack on the US Capitol, listened to Trump's call with Brad Raffensperger, and The Trump Tapes with Bob Woodward, and Liz Cheney's Oath and Honor.
Different table top roll playing game systems
I fell into the rabbit hole of delinking programs back into object files.
Long story short, I was inspired by the Super Mario 64 and REDRIVER2 decompilation projects and wanted to do one. I picked a PlayStation video game from my childhood, started Ghidra and then I quickly realized that the game code's a complete mess. It's bad enough that I don't see myself ever finishing this project unless I can somehow divide-and-conquer this problem into manageable pieces. But you can't exactly break a program into pieces... can you?
So I've started to think for a bit and remembered the basic toolchain workflow: source files are compiled into assembly files, which are assembled into object files, which are all linked together into a program. The last bit stood out to me and I wondered: what if I could undo the work of the linker? I'd get a bunch of object files, dividing the original reverse-engineering problem into smaller pieces.
I searched online and found absolutely nothing on the topic. That should've tipped me off, but instead I started scribbling on a piece of paper. Object files are made up of sections (named arrays of bytes), symbols (named offsets within these sections) and relocations (spots to patch with a symbol's address). The linker lays out the sections, computes the addresses of the symbols and then patches the relocation spots to produce the program. I can't just take the program bytes and stuff them into object files because of these applied relocations, but if I could somehow undo them...
The good idea fairy struck, and the fairy struck hard.
I'm writing scripts in Jython and after a couple hundred lines I get results on sample test cases. I try them on the game and it takes forever due to algorithmic complexity. I rewrote a new implementation in Java, forking Ghidra in the process. I rewrote it a couple more times because my analyzer kept hitting edge cases. I built an elaborate and exhaustive test harness because I keep introducing hard to track down regressions. I submitted a couple of pull requests to Ghidra to solve some painful points and reduce the size of the diff, which spanned thousands of lines. I reply to the questions from the Ghidra team with walls of texts trying to explain my use-case, but the PRs get rejected because they don't fit the current design of Ghidra well.
When the Ghidra team rejected my stuff probably because at this point I was probably speaking in the native language of Cthulhu, I really should've taken the hint.
Instead, I spin off my fork as a Ghidra extension to alleviate the maintenance burden, which by now was getting closer to ten thousand lines. I keep rewriting my MIPS relocation analyzers again and again to improve their correctness, always hitting a new edge case. I've decided to start a blog, because I'm tired of trying to explain this stuff from basic principles to people since there's no literature on this topic. I get side-tracked writing a complete series of articles on the basics of reverse-engineering to introduce the topic. I get side-tracked again writing a series of articles on the applications of delinking related to software ports, with a case study on a x86 program that requires me to write relocation analyzers for this architecture and perform refactorings to support multiple ISAs and object file formats.
I'm finally back on reverse-engineering the video game that started all of this and get side-tracked once more because I'm documenting the process in another series of articles. By sheer luck I stumble upon a SYM debugging symbols file, but I don't have the matching executable for it, so I build a placeholder one that matches its shape, then import the placeholder into Ghidra, then write about a thousand lines of Java to import this data on top of the placeholder, then write a bunch of scripts and my own correlators to version track it onto a executable I do have because Ghidra doesn't know what to do with a source executable that doesn't have a single initialized byte to its name. I've tried to engage with the Ghidra community about this latest problem, but no answer. I assume they're probably busy trying to find an exorcist, so I carry on regardless.
Two years. Two years I've spent digging this rabbit hole that's probably worth a thesis or two. I know enough about delinking now that I could probably write a book that would read like a Lovecraftian horror story to people that develop linkers for a living. I've automated this stuff down to making a selection inside Ghidra and clicking on "File > Export Program...", but there's only so much you can do to make accessible or even understandable a technology that allows you to literally rip out code from a Linux program and shove into a Windows program or from a PlayStation game into a Linux program and have it work, in spite of ABIs or common sense.
TL;DR I've developed a reverse-engineering technique by accident that would give professors teaching Computer Sciences 101 an existential crisis.
FOUNTAIN PENS
vtuber?
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
So, I decided to install Linux on my formerly-Windows-only laptop, and thought it was cool enough to go full time and ditch Windows completely. The downside was the lack of access to top tier games. No problem though, my plan was to take a break from gaming, figuring that by the time Linux had caught up with compatibility, computers would also be much more powerful and I'd be able to resume gaming at some point in the future on better kit, and not have to worry about janky framerates on struggling hardware.
Linux proved interesting enough that I kept finding all sorts of cool new rabbit holes to go down - shell scripting, filesystems, Python, databases. It was side-quests within side-quests! Plus, having kicked my gaming habit, I had plenty of time to explore these.
Anyway, to cut a long story short, that was 23 years ago. I ended up getting a career in tech, relocated, got married, had kids, lived the American Dream... The "life" rabbit hole kind of got in the way of my plans, so I can't wait to finally get back on track and play GTA III on a decent box.