Huh. I hate the streaming mode of ChatGPT. In my experience, any UI that adopts it ends up feeling slow and sluggish for no apparent reason. Plus, I often find myself looking at the text as it appears, and wondering if I should press the "Stop" button, or will it say something useful in the two paragraphs I expect to still follow.
Alas, I use the streaming mode, because it gives an indicator that the connection to the model is still live, and you can kind of estimate how much longer you'll have to wait for the reply to end. With "batched" mode, you just sit there for anywhere between few seconds and few minutes, wondering if it's still generating response, or if the API connection broke again.
When it streams too slowly it is also quite annoying, but when it is the right speed I really enjoy it.
It can also be annoying when I made request a small change and it rewrites the entire block of code compared to only displaying the section I want. I think I could prompt it better in those cases by saying "only show me the code changes" though.
I'm working on relevant prompts. I've managed to get it to output edits a few times, but most of the time it does ignore my requests to just give me the changes.
I think it's mostly my laziness. I found GPT-4 with default system message good enough for most use cases, so I tend to just switch over to it, start a new conversation, and blurt my request. Setting specialized "characters" for my requests? Too much hassle. Typing in a proper, explicit request to just give me diffs? Too much writing, breaks flow.
Ironically, I'm both too lazy for giving GPT-4 this kind of editing guidance, while I'm also wasting too much time catering to conversational style.
Sad part is that I think it would be quite easy to make a handful of UIs with the built in system prompts to achieve exactly what you want... And someone is going to do it and charge money for it.
> And someone is going to do it and charge money for it.
This already happened. It's called TypingMind.com; I'm happy paying user. It's main selling point is that it isn't dog slow and isn't trying to take down my browser with its bloat, the way OpenAI chat interface is. "Character" creation and selecting is lovely, but half the time I'm too lazy to switch away from the default.
It sounds like you just want a faster response, not a batched one, given that you don't use the batched one for the same reason we prefer the streaming one?
Just imagine it's a human on the other side typing or speaking, and you're getting to hear / see what they type as they type it.
It is, in fact, extremely strange for humans to have the experience of sitting waiting for a complete utterance before processing: it feels more like sending a letter than talking.
There were even chat systems that worked like this, including (I think?) the Unix talk command and one of the popular ICQ-era chat apps (maybe it was ICQ itself?).
> Just imagine it's a human on the other side typing or speaking, and you're getting to hear / see what they type as they type it.
Yes. I imagine it and I shudder from suddenly experiencing how annoying that would be.
> There were even chat systems that worked like this, including (I think?) the Unix talk command
Correct. I've used `talk` in high school (and a colorized variant called `ytalk.qq`). It was a fun experience, and I miss it a bit - since then, the only software I used that kind of reproduced it, was Etherpad.
Now, the way `talk` conversations and real-life conversations are tolerable, whereas watching someone (a human or a bot) typing 'till completion is frustrating, is in the speed of the feedback loop. When talking (or `talk`ing), I can interrupt you when I see you saying things I already know. You can interrupt me too. This happens at the level of phonemes/characters, and allows the conversation to flow dynamically - one moment, we'd have a rapid back-and-forth, another moment, I'd listen to you very carefully, etc.
With normal messengers of today, if I were to see you typing (instead of having the - also annoying, but for other reasons[0] - pencil icon), but had no way to interrupt you - I'd be frustrated all the time, being able to know with good confidence what you want to say while you're 1/3rd through your message. Natural language, especially in conversations, is stupidly redundant. I'd just Alt+TAB to HN or something, and wait for you to finish[1].
This is my experience with GPTs. I often want them to just stop typing, but if I hit "Stop", I might just break the response in the wrong moment, creating cleanup work for myself. The side effect of the streaming chat UIs being dog slow makes this even worse: there's no telling how many seconds it'll take for my "Stop" to register. For some reason, this gets both worse and less predictable as the conversation gets longer.
--
[0] - All kinds of interpersonal nonsense, but a big factor is that, with a bot, you know it's actually generating a message and will finish soon (unless the connection fails). With humans? Who the hell knows. They may type you an essay. Or type two sentences for 5 minutes. Or get distracted by cat. Or just be screwing with you. With humans, half the time I'm considering looking for ways to disable typing indicators.
[1] - Which is exactly what I do most the time with people and typing indicators. And GPTs.
Alas, I use the streaming mode, because it gives an indicator that the connection to the model is still live, and you can kind of estimate how much longer you'll have to wait for the reply to end. With "batched" mode, you just sit there for anywhere between few seconds and few minutes, wondering if it's still generating response, or if the API connection broke again.