One of my favorite tests for chatbots is asking for book recommendations. I give it a list of books I liked and books I didn't like (and some flavor for why) and ask them what to read.
They're... ok at this, mostly. It's funny because I always feel like this should be a very straightforward traditional ML problem to do with Goodreads data or whatever but none of the things which purport to be that (Storygraph, etc) are any good at all.
Anyway, o3-mini seems to be the best at this so far for whatever reason. With the same prompt as I've been using elsewhere, it gave me 7 books of which I'd already read and enjoyed 5. Best hit rate on that metric from other chatbots was ~1/4, and in several cases they included books in a series I'd explicitly said as part of the prompt that I didn't enjoy.
like this
Kevin Gibbons
in reply to Kevin Gibbons • •My prompt, incidentally:
... show moreMy prompt, incidentally:
Kevin Gibbons
in reply to Kevin Gibbons • •And o3's response:
... show moreAnd o3's response:
(I've read and enjoyed Dirk Gently, Good Omens, Soon I Will Be Invincible, Redshirts, and The Eyre Affair.
Ben Weinstein-Raun likes this.
Kevin Gibbons
in reply to Kevin Gibbons • •Claude Sonnet 3.7 and GPT 4.5 both do pretty well by this metric.
Oddly, they both recommended Senlin Ascends, which no previous model has mentioned. It's from 2013, so it's not like it wasn't in the training data for the other models. I guess I'll have to try it!
3.7 is also the first to recommend Terra Ignota (Too Like the Lightning), which are I think my favorite books I've ever read. (I didn't put them in the prompt because they're pretty weird and most books which are that weird are not to my taste.)
Ben Weinstein-Raun likes this.
Kevin Gibbons
in reply to Kevin Gibbons • •I also appreciated this from Claude:
> Ender's Game by Orson Scott Card - Strategic thinking protagonist if you haven't read it already
... Yes, Clause, you're absolutely correct to assume that I've probably already read Ender's Game based on the list of books I enjoyed, well done.
Kevin Gibbons
in reply to Kevin Gibbons • •On a completely different dimension, GPT4.5 is also the best at writing poetry of any model I've tried. I give models Afton Water and ask them to
> write a few more stanzas suitable for an urban environment. Be sure to preserve the rhyme scheme and pay attention to the scansion. After writing, analyze how you did on rhyme and rhythm and make modifications (potentially substantial modifications) as necessary, and repeat. It's OK for your response to be very long, with multiple iterations - that's how writing goes for humans too.
Thus far, all attempts from other models have failed to scan. The previous ChatGPT gave me "Ye sighing train-whistles from side-streets that flow / Soft rustling branches in sycamores fair" for example. This is kind of unfair because scansion is very hard to derive for a model which is trained on text, but it's still something I care about. GPT4.5 pretty much nails the scansion except in one place ("lamps casting" is two stressed s
... show moreOn a completely different dimension, GPT4.5 is also the best at writing poetry of any model I've tried. I give models Afton Water and ask them to
> write a few more stanzas suitable for an urban environment. Be sure to preserve the rhyme scheme and pay attention to the scansion. After writing, analyze how you did on rhyme and rhythm and make modifications (potentially substantial modifications) as necessary, and repeat. It's OK for your response to be very long, with multiple iterations - that's how writing goes for humans too.
Thus far, all attempts from other models have failed to scan. The previous ChatGPT gave me "Ye sighing train-whistles from side-streets that flow / Soft rustling branches in sycamores fair" for example. This is kind of unfair because scansion is very hard to derive for a model which is trained on text, but it's still something I care about. GPT4.5 pretty much nails the scansion except in one place ("lamps casting" is two stressed syllables in a row), and the imagery is better too. Also it made it about the flow of a city instead of the flow of a river, which works way better in an urban setting.
> Flow softly, fair city, beneath starlit skies,
> Where towers of silver and glass gently rise;
> My Mary's asleep by thy lantern's calm beam—
> Flow softly, dear city, disturb not her dream.
>
> Ye engines that murmur through streets in the night,
> Soft glow of the lamps casting mellowing light;
> Thou bell from the chapel, be quiet this hour,
> Disturb not my Mary asleep in her bower.
>
> How graceful thy buildings, illumined and clear,
> That glisten past twilight as evening draws near;
> There daily I wander down avenues fair,
> My Mary's dear dwelling my joy and my care.
>
> How lovely thy parks as they stretch far below,
> With lilacs and roses that softly do grow;
> There oft at the dusk I sit calm and carefree,
> On elm-shaded benches, my Mary and me.
>
> Thy streets and thy alleys, how gently they wind,
> Beyond glowing windows where Mary reclined;
> The city sleeps quiet, my footsteps are light,
> To guard gently Mary, asleep through the night.
Ben Weinstein-Raun likes this.