While somewhat useful - there are no systematic ablation and comparison studies, datasets, and quantitative evals here.
It's all anecdotal, so should probably be a blogpost and not a "paper"
"systematic ablation" - https://stats.stackexchange.com/questions/380040/what-is-an-.... How do you ablate a black box? You work with the inputs. I agree that this article lacks rigor, but I don't mind some structured thinking about what seems to work. I don't think the authors over-promised here.
>List all the entities in the text that are ambiguous. Entities are ambiguous if you don't understand them, if they have multiple meanings and you can't decide which one from the context, or are phraseal terms whose meaning can be different in different cultures.
This allows the LLM to identify parts of previous prompt that the AI did not fully understand.
>List all the entities and relationships to other entities. Be precise, avoid duplicates, list each relationship only once.
This allows to compress the past context for longer tasks.
>Make a list of Google searches needed to build a factual answer, one sentence per search.
You get the list then
>For each sentence extract the main entity, decide whether to find more information about it on Google, wikipedia, ..., answer in json
This is my "ghetto" agent, it's not iterative, but it's stable enough to handle the unexpected
> It's the context enough to answer factually the user question? Answerer yes or no. Only answer yes if you are sure you can answer
To make the agent able to loop, or ask user for more context.
It's maybe interesting to think about a simple chat history over time giving some of the knowledge needed for improved interactions as related to the more complex patterns outlined here.
I indexed both documents with my own project, which uses semantic graphs to help with prompt assembly: https://github.com/FeatureBaseDB/DoctorGPT. DoctorGPT doesn't have dynamic prompt chaining yet, but I'm working on it. I hesitate posting any of the analysis of these papers using DoctorGPT here because it would be generated by the LLM, and not me...and some people seem to have an issue with that given this is a human forum.
My sense is that SKGs are important in refining questions, offering alternative approaches, managing context, reflecting on LLM responses, and more.
I really like the troubleshooting advice for the installation of DoctorGPT:"These instructions are long, so ensure you follow them carefully. It is suggested you use ChatGPT to assist you with any errors. Simply paste in the entire content of this README into ChatGPT before asking your question about the install process."
I think you shouldn't call it "prompt engineering" until you do some real "engineering". For example, if you use Guidance from microsoft, where you write a prompt "template" and let guidance filter the logits probabilities at the corresponding points for you, I claim that this is "engineering (the way we use it as SWEs anyway) because it's analogous to writing an actual program.
I was actually upset about everyone claiming they're a "prompt engineer" without any real engineering that I wrote a snarky github gist about it that was on the front page for a big. It's mainly pointing out that NLP can't even do real prompt engineering like Stable Diffusion folks can because we didn't build the right tooling for it yet.
I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent.
I can't wait until there is a standardized ChatGPT Bible to guide this emerging machine-priest caste. And these posts will be the equivalent of the dead sea scrolls.
Have you tried getting great, repeatable results out of an LLM? It requires great depth of knowledge - about both how LLMs work and the specific topic you are trying to build against - plus a methodical process in figuring out what works and what doesn't.
I see no reason not to label that "engineering".
I also think it's important to distinguish between prompting and prompt engineering. Prompting is when people type prompts in a box. Prompt engineering, by my own definition, is when developers build further software on top of LLMs.
Engineering is the application of science; as the science here is weak at best, calling it engineering seems a stretch. Perhaps “prompt author” would be more appropriate.
I think that there is a science here that people are working to discover. Prompt engineering is slowly turning into its own language and the more it's studied like this I think the more validity there will be to calling it engineering.
Given the term 'software engineer' is now almost universally accepted (outside of legal jurisdictions where engineering = liability) I don't see why prompt engineering is a bad term.
