Category: coding

Posted on

TokenFlow: Visualize LLM token streaming speeds

Have you ever wondered how fast your favorite LLM really compares to other SoTA models? I recently saw a Reddit post where someone was able to get a distilled version of Deepseek R1 running on a Raspberry Pi! It could generate output at a whopping 1.97 tokens per second. That sounds slow. Is that even usable? I don’t know!

Meanwhile, Mistral announced that their Le Chat platform can output tokens at 1,100 per second! That sounds pretty fast? How fast? I don’t know!

So, that’s why I put together TokenFlow. It’s a (very!) simple webpage that lets you see the speed of different LLMs in action. You can select from a few preset models / services or enter a custom speed, and boom! You watch it spit out tokens in real time, showing you exactly how fast a given inference speed is for user experience.

Check it out: https://dave.ly/tokenflow/

The code is also available on Github.

Posted on

Using AI: Extracting reading list from GoodReads

I feel like I want to start creating some posts related around how I, as a software engineer, personally use generative AI tools. I think they are a huge boon for increasing productivity, exploring new ideas, and even learning new things

Reading Reddit, Hacker News, and various other forums, there’s a lot of anxiety among software engineers about how AI is going to steal our jobs. It’s not without merit:

A recent blog post by Dustin Ewers adds some needed sanity to the discussion. In a post titled, “Ignore the Grifters – AI Isn’t Going to Kill the Software Industry“, he argues that:

He argues that we should ignore the grifters.

I feel like half of my social media feed is composed of AI grifters saying software developers are not going to make it. Combine that sentiment with some economic headwinds and it’s easy to feel like we’re all screwed. I think that’s bullshit. The best days of our industry lie ahead.

It’s highly unlikely that software developers are going away any time soon. The job is definitely going to change, but I think there are going to be even more opportunities for software developers to make a comfortable living making cool stuff.

I am inclined to agree. Hey, I will drink this Kool-Aid!

—

Well, let’s get to the real reason I’m making this post. I posted about my 2024 reading list and shared all the books I had read during the year. Try to compile that list and add links by hand would be a huge pain. There has to be an easier way. (Cue super hero music)

There is!

If you go to my GoodReads “read” list, it looks like this. And it keeps going. It’s a lot of data.

If we open up the browser console, we can see that it’s just a good old fashioned HTML table.

So, using an AI tool like ChatGPT or Claude, how do you get this data in a manageable way? One area that I’ve personally seen people struggle with is how to write a prompt in a way that helps them. You need to provide context. You say:

  1. Describe the problem: “I want to output a list of books from an HTML table into a JSON object using a JavaScript function that I can paste into the browser console.”
  2. Provide some example date: “Here is the table’s head with names for each column: [paste block of code]. Oh! Here is also an example row of data: [paste block of code]”
  3. Provide an example of the output: “Can you create a JSON object in the following shape?”

Using Claude as an example, here is what that looks like and you can also see the generated output:

Moment of truth — does it work? Let’s paste it into the browser console and see the result:

Yes! Victory! One problem though. I did not read 60 books in 2024. Oh, no. We are pulling all books visible from the page. This isn’t a problem. We can fix it by simply asking a followup question: “Can we modify the function so that it only returns books where date read is in the year 2024?”

Claude modifies the function to add a filter for 2024. If we paste that into the browser console, we now get the correct number of hooks: 30!

There is still another thing to do. I want to make this into a nice, unordered list that I can just add into my blog post. Again, we follow the steps outlined above:

  1. Can you create an unordered HTML list that shows links to each book? Please add a link around the title, but keep the author name unlinked.
  2. Here is my JSON object: [paste block of code]
  3. I essentially want a list that looks like this: <li><a href=”[booklink”>Book title</a> by Author</li>

Hot diggity! It works. It generates a block of code that I can just paste into my blog’s text editor. Pretty neat. It took a total of 5 minutes. (Hey, writing this post took a lot longer than that.)

Anyway, this has been a production of “How I use AI”. Stay tuned for more exciting updates, coming to a blog near you.

Posted on

Project: Super Simple ChatUI

I’ve been playing around a lot with Ollama, an open source project that allows one to run LLMs locally on their machine. It’s been fun to mess around with. Some benefits: no rate-limits, private (e.g., trying to create a pseudo therapy bot, trying to simulate a foul mouthed smarmy sailor, or trying to generate ridiculous fake news articles about a Florida Man losing a fight to a wheel of cheese), and access to all sorts of models that get released.

I decided to try my hand at creating a simplified interface for interacting with it. The result: Super Simple ChatUI.

As if I need more side projects. So it goes!

Posted on

TIL: List git branches by recent activity

In both my work and personal coding projects, I generally have a number of various branches going at once. Switching between various branches (or remembering past things I was working on) can somethings be a chore. Especially if I’m not diligent about deleting branches that have already been merged.

Usually, I do something like:

> git branch

Then, I get a ridiculously huge list of branches that I’ve forgotten to prune and spend all sorts of time trying to remember what I was most recently working on.

daves/XXXX-123/enable-clickstream
daves/XXXX-123/impression-events
daves/XXXX-123/tracking-fixes
daves/XXXX-123/broken-hdps
daves/XXXX-123/fix-contacts
daves/XXXX-123/listing-provider
daves/XXXX-123/revert-listing-wrapper-classname
daves/XXXX-123/typescript-models
daves/XXXX-123/inline-contact-form
daves/XXXX-123/clickstream_application_event
daves/XXXX-123/unused-file
daves/XXXX-123/convert-typescript
daves/XXXX-123/convert-typescript-v2
daves/XXXX-123/similar-impressions
daves/XXXX-123/update-node-version

At least 75% of those have already been merged and should have been pruned.

There has to be a better way, right?

Thanks to the power of the Google machine (and Stack Overflow), I found out, there is!

> git branch --sort=-committerdate

Hot diggity dog!

daves/XXXX-123/clickstream-filter-events
main
daves/XXXX-123/convert-typescript-v2
daves/XXXX-123/update-node-version
daves/XXXX-123/similar-impressions
daves/XXXX-123/convert-typescript
daves/XXXX-123/clickstream_application_event
daves/XXXX-123/unused-file
daves/XXXX-123/typescript-models
daves/XXXX-123/listing-provider
daves/XXXX-123/inline-contact-form
daves/XXXX-123/revert-listing-wrapper-classname

That list is now sorted by most recent activity on the branch.

Alright. Even though this is better, that’s still a lot of typing to remember. Fortunately, we can create an alias:

> git config --global alias.recent "branch --sort=-committerdate"

Now all I need to do is just type git recent and it works!

Nice.

Posted on

TIL: How to change your default editor for git commits

A recent post on Hacker News highlighted the benefits of detailed commit messages in git.

Usually, my git commits look something like this:

> git commit -m "fix: component missing configuration file"

…which isn’t all that helpful. (Related: see XKCD on git commit messages)

I decided to try and utilize this newfound knowledge in my own git commits and I quickly ran into an obstacle. Simply using > git commit opens up vim. Which, I really don’t want to use. (I’m sorry!)

This is something I should already know how to do, but I had to do a Google search to learn more. It turns out, you can change the default editor in git. This makes it much more convenient! How do you do it?

git config --global core.editor "nano"

Replace “nano” with your preferred editor of choice. Now, running > git commit opens up your editor and you can make detailed commit messages to your heart’s content!

Posted on

Upgrading Mr. RossBot’s image model and prompt template

My Mastodon landscape painting bot, Mr. RossBot keeps kicking along, generating some fun landscape art. It’s been powered by the AI Horde (the open source project behind ArtBot) and has tried to utilize whatever image models provided by the API to the best of its abilities.

For the most part, the code behind it is a bunch of spaghetti that looks like this:

An update to the AI Horde late last year added support for SDXL. However, the SDXL model on the Horde did not use a refiner. Because of this, images tended to come out a bit soft and lacked texture.

You can see examples of this in my announcement post about Mr. RossBot being back, here. See also:

More recently, the Horde added support for a new image model: AlbedoBaseXL. It’s an SDXL model that has a refiner baked in. Now images will come out a lot sharper looking.

Coincidentally, I was also playing around with various prompts and discovered I could get much better image results that look more painterly (rather than simple digital renderings) by utilizing the following prompt:

A beautiful oil painting of [LITERALLY_ANYTHING], with thick messy brush strokes.

And that is it! No more messy appending various junk to the end of the prompt to attempt to get what I want. The results speak for themselves and are pretty awesome, I think!

Posted on

Implementing and testing a “poor man’s prompt expansion” model for Stable Diffusion

Various Stable Diffusion models massively benefit from verbose prompt descriptions that contain a variety of additional descriptors. Much recent research has gone into training text generation models for expanding existing Stable Diffusion prompts with relevant and context appropriate descriptors.

Since it isn’t feasible to run LLMs and text generation models inside most users’ web browsers at this time, I present my “Poor Man’s Prompt Expansion Model“. It uses a number of examples I’ve acquired from Fooocus and Hugging Face to generate completely random (and absolutely not context appropriate) prompt expansions.

(For those interested in following along at home, you can checkout the gist for this script on GitHub).

How does it work?

We iterate through a list of an absolute crap ton of prompt descriptors that I’ve sourced from other (smarter) systems that tokenize user prompts and attempt to come up with context appropriate responses. We’re not going to do that, because we’re going to go into full chaos mode:

  1. Iterate through a list of source material and split up everything separated by a comma.
  2. Add the resulting list to a new 1-dimensional array.
  3. Now, build a new descriptive prompt by looping through the list until we get a random string of descriptors that are between 175 and 220 characters long.
  4. Once that’s done, return the result to the user.
  5. Create a new prompt.

For our experiment, we’re going to lock all image generation parameters and seed, so we theoretically get the same image given the exact same parameters.

Ready?

Here is our base prompt and the result:

Happy penguins having a beer

Not bad! Now, let’s go full chaos mode with a new prompt using the above rules and check out the result:

Happy penguins having a beer, silent, 4K UHD image, 8k, professional photography, clouds, gold, dramatic light, cinematic lighting, creative, pretty, artstation, award winning, pure, trending on artstation, airbrush, cgsociety, glowing

That’s fun! (I’m not sure what the “silent” descriptor means, but hey!) Let’s try another:

Happy penguins having a beer, 8k, redshift, illuminated, clear, elegant, creative, black and white, masterpiece, great power, pinterest, photorealistic, award winning, vray, enchanted, complex, excellent composition, beautiful composition

I think we just created an advertisement for a new type of beverage! It nailed the “black and white”, though I’m not sure how that penguin turned into a bottle. What else can we make?

Happy penguins having a beer, volumetric lighting, Digital, intricate, awesome, futuristic, cartoon artstyle, vector, solid, detailed, dramatic light, realistic photograph, wonderful colors, dramatic atmosphere

The dude in the middle is planning on having a good night. Definitely some “wonderful colors”. Not so much realistic photo or vector, but fun! One last try:

Happy penguins having a beer, 35mm, surreal, amazing, Trending on Artstation HQ, matte painting hyperrealistic, full focus, very inspirational, pixta.jp, aesthetic, 8k, black and white, reflected on the matrix studio background, awesome

As you can see, you can get a wide variety of image styles by simply mixing a bunch of descriptive elements to an image prompt.

I’ve wanted to implement a feature like this on ArtBot for a long time. (Essentially, if the user allows it, automatically append these descriptions behind the scenes when an image is requested). Perhaps this will come soon.

Posted on

DALL-E 3: Adding text to your text-to-prompt images

I recently got access to DALL-E 3 through OpenAI’s ChatGPT+ interface. One of the key features and improvements in their image model is the ability to generate coherent text within the image.

Let’s give it a try, based on one of the most popular StackOverflow questions: How do I exit Vim?

Using the following prompt: Oil painting of a hacker furiously typing commands into an old computer and muttering to himself, “how does one exit vim?”

That… is pretty good!