Tootfinder

The term "context engineering" is gaining traction over "prompt engineering" as it better describes the skill of providing LLMs with the necessary information (Simon Willison/Simon Willison's Weblog)
https://simonwillison.net/2025/Jun/27/context-…

Documenting Your Prompts a Best Practice for Success
#promptengineering

Re-Evaluating Code LLM Benchmarks Under Semantic Mutation
Zhiyuan Pan, Xing Hu, Xin Xia, Xiaohu Yang
https://arxiv.org/abs/2506.17369 https://

Re-Evaluating Code LLM Benchmarks Under Semantic Mutation
In the era of large language models (LLMs), code benchmarks have become an important research area in software engineering and are widely used by practitioners. These benchmarks evaluate the performance of LLMs on specific code-related tasks, such as code understanding and generation. A critical step in constructing code benchmarks is the design of prompts. However, as existing code benchmarks typically rely on a single prompt template per task, they are prone to the issue of prompt sensitivity…

Struggling with generic or off-target AI outputs? 🤔 Mastering how you ask – known as Prompt Engineering – is key to unlocking truly useful responses from tools like Gemini and ChatGPT.
Simple shifts, like providing examples or guiding the AI's reasoning, make a huge difference. ✨ I've put together a straightforward video guide covering these foundational techniques (think few-shot, chain of thought, setting context & more) to help elevate your AI interactions.
🎥:

Doppelg\"anger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack
Daewon Kang, YeongHwan Shin, Doyeon Kim, Kyu-Hwan Jung, Meong Hi Son
https://arxiv.org/abs/2506.14539

Doppelgänger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack
Since the advent of large language models, prompt engineering now enables the rapid, low-effort creation of diverse autonomous agents that are already in widespread use. Yet this convenience raises urgent concerns about the safety, robustness, and behavioral consistency of the underlying prompts, along with the pressing challenge of preventing those prompts from being exposed to user's attempts. In this paper, we propose the ''Doppelgänger method'' to demonstrate the risk of an agent being hij…

Optimizing Web-Based AI Query Retrieval with GPT Integration in LangChain A CoT-Enhanced Prompt Engineering Approach
Wenqi Guan, Yang Fang
https://arxiv.org/abs/2506.15512

Optimizing Web-Based AI Query Retrieval with GPT Integration in LangChain A CoT-Enhanced Prompt Engineering Approach
Large Language Models have brought a radical change in the process of remote learning students, among other aspects of educative activities. Current retrieval of remote learning resources lacks depth in contextual meaning that provides comprehensive information on complex student queries. This work proposes a novel approach to enhancing remote learning retrieval by integrating GPT-based models within the LangChain framework. We achieve this system in a more intuitive and productive manner using…

Prompt Engineering kam und ging
https://stohl.de/wordpress/?p=191223

Prompt Engineering kam und ging
Prompt Engineering sollte ein neues Berufsbild werden und dennoch ist der Hinweis aus Stellenbeschreibungen 2025 praktisch vom Arbeitsmarkt verschwunden: Prompt-Ingenieure werden nicht mehr gebrauc…

Automatic Large Language Models Creation of Interactive Learning Lessons
Jionghao Lin, Jiarui Rao, Yiyang Zhao, Yuting Wang, Ashish Gurung, Amanda Barany, Jaclyn Ocumpaugh, Ryan S. Baker, Kenneth R. Koedinger
https://arxiv.org/abs/2506.17356

Automatic Large Language Models Creation of Interactive Learning Lessons
We explore the automatic generation of interactive, scenario-based lessons designed to train novice human tutors who teach middle school mathematics online. Employing prompt engineering through a Retrieval-Augmented Generation approach with GPT-4o, we developed a system capable of creating structured tutor training lessons. Our study generated lessons in English for three key topics: Encouraging Students' Independence, Encouraging Help-Seeking Behavior, and Turning on Cameras, using a task deco…

The word "hacker" has been hijacked to mean something even worse than "cracker" (from LinkedIn):

Last leg on our brief history of NLP (so far) is the advent of large language models with GPT-3 in 2020 and the introduction of learning from the prompt (aka few-shot learning).
T. B. Brown et al. (2020). Language models are few-shot learners. NIPS'20
https://…

Can Generated Images Serve as a Viable Modality for Text-Centric Multimodal Learning?
Yuesheng Huang, Peng Zhang, Riliang Liu, Jiaqi Liang
https://arxiv.org/abs/2506.17623

Can Generated Images Serve as a Viable Modality for Text-Centric Multimodal Learning?
A significant ``modality gap" exists between the abundance of text-only data and the increasing power of multimodal models. This work systematically investigates whether images generated on-the-fly by Text-to-Image (T2I) models can serve as a valuable complementary modality for text-centric tasks. Through a comprehensive evaluation framework on text classification, we analyze the impact of critical variables, including T2I model quality, prompt engineering strategies, and multimodal fusion arch…

Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
Daniel D'souza, Julia Kreutzer, Adrien Morisot, Ahmet \"Ust\"un, Sara Hooker
https://arxiv.org/abs/2506.14702

Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
One of the most profound challenges of modern machine learning is performing well on the long-tail of rare and underrepresented features. Large general-purpose models are trained for many tasks, but work best on high-frequency use cases. After training, it is hard to adapt a model to perform well on specific use cases underrepresented in the training corpus. Relying on prompt engineering or few-shot examples to maximize the output quality on a particular test case can be frustrating, as models …

Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks
E. G. Santana Jr, Gabriel Benjamin, Melissa Araujo, Harrison Santos, David Freitas, Eduardo Almeida, Paulo Anselmo da M. S. Neto, Jiawei Li, Jina Chun, Iftekhar Ahmed
https://arxiv.org/abs/2506.05614

Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks
A growing variety of prompt engineering techniques has been proposed for Large Language Models (LLMs), yet systematic evaluation of each technique on individual software engineering (SE) tasks remains underexplored. In this study, we present a systematic evaluation of 14 established prompt techniques across 10 SE tasks using four LLM models. As identified in the prior literature, the selected prompting techniques span six core dimensions (Zero-Shot, Few-Shot, Thought Generation, Ensembling, Sel…

Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs
Nariman Naderi, Zahra Atf, Peter R Lewis, Aref Mahjoub far, Seyed Amir Ahmad Safavi-Naini, Ali Soroush
https://arxiv.org/abs/2506.00072

Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs
This paper investigates how prompt engineering techniques impact both accuracy and confidence elicitation in Large Language Models (LLMs) applied to medical contexts. Using a stratified dataset of Persian board exam questions across multiple specialties, we evaluated five LLMs - GPT-4o, o3-mini, Llama-3.3-70b, Llama-3.1-8b, and DeepSeek-v3 - across 156 configurations. These configurations varied in temperature settings (0.3, 0.7, 1.0), prompt styles (Chain-of-Thought, Few-Shot, Emotional, Exper…

Enhancing Finite State Machine Design Automation with Large Language Models and Prompt Engineering Techniques
Qun-Kai Lin, Cheng Hsu, Tian-Sheuan Chang
https://arxiv.org/abs/2506.00001

Enhancing Finite State Machine Design Automation with Large Language Models and Prompt Engineering Techniques
Large Language Models (LLMs) have attracted considerable attention in recent years due to their remarkable compatibility with Hardware Description Language (HDL) design. In this paper, we examine the performance of three major LLMs, Claude 3 Opus, ChatGPT-4, and ChatGPT-4o, in designing finite state machines (FSMs). By utilizing the instructional content provided by HDLBits, we evaluate the stability, limitations, and potential approaches for improving the success rates of these models. Further…

Prompt Engineer: Analyzing Skill Requirements in the AI Job Market
An Vu, Jonas Oppenlaender
https://arxiv.org/abs/2506.00058 https://

Prompt Engineer: Analyzing Skill Requirements in the AI Job Market
The rise of large language models (LLMs) has created a new job role: the Prompt Engineer. Despite growing interest in this position, we still do not fully understand what skills this new job role requires or how common these jobs are. We analyzed 20,662 job postings on LinkedIn, including 72 prompt engineer positions, to learn more about this emerging role. We found that prompt engineering is still rare (less than 0.5% of sampled job postings) but has a unique skill profile. Prompt engineers ne…

This https://arxiv.org/abs/2405.08965 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csPL_…

Meaning-Typed Programming: Language Abstraction and Runtime for Model-Integrated Applications
Software development is shifting from traditional logical programming to model-integrated applications that leverage generative AI and large language models (LLMs) during runtime. However, integrating LLMs remains complex, requiring developers to manually craft prompts and process outputs. Existing tools attempt to assist with prompt engineering, but often introduce additional complexity. This paper presents Meaning-Typed Programming (MTP) model, a novel paradigm that abstracts LLM integratio…

A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming
Yizhong Ding
https://arxiv.org/abs/2505.24252 https://arx…

A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming
Frequent cyber-attacks have elevated WebShell exploitation and defense to a critical research focus within network security. However, there remains a significant shortage of publicly available, well-categorized malicious-code datasets organized by obfuscation method. Existing malicious-code generation methods, which primarily rely on prompt engineering, often suffer from limited diversity and high redundancy in the payloads they produce. To address these limitations, we propose \textbf{RAWG}, a…

Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
Anirudh Nair, Adi Banerjee, Laurent Mombaerts, Matthew Hagen, Tarik Borogovac
https://arxiv.org/abs/2506.00178

Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
Prompt engineering represents a critical bottleneck to harness the full potential of Large Language Models (LLMs) for solving complex tasks, as it requires specialized expertise, significant trial-and-error, and manual intervention. This challenge is particularly pronounced for tasks involving subjective quality assessment, where defining explicit optimization objectives becomes fundamentally problematic. Existing automated prompt optimization methods falter in these scenarios, as they typicall…

Our colleague Ebrahim Norouzi is presenting ConExion: Concept Extraction with Large Language Models, yoint work with Sven Hertling. Comprehensive Prompt Engineering Study of LLM-based concept extraction for domain coverage evaluation of ontologies.
arxive version: https://arxiv.org/abs/2504.12915…

ConExion: Concept Extraction with Large Language Models
In this paper, an approach for concept extraction from documents using pre-trained large language models (LLMs) is presented. Compared with conventional methods that extract keyphrases summarizing the important information discussed in a document, our approach tackles a more challenging task of extracting all present concepts related to the specific domain, not just the important ones. Through comprehensive evaluations of two widely used benchmark datasets, we demonstrate that our method improv…

This https://arxiv.org/abs/2506.00490 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csNE_…

LLM-Driven Instance-Specific Heuristic Generation and Selection
Combinatorial optimization problems are widely encountered in real-world applications. Designing high-quality heuristic algorithms that efficiently approximate optimal solutions within reasonable time is a critical research challenge. In recent years, many works have explored integrating Large Language Models (LLMs) with Evolutionary Algorithms to automate heuristic algorithm design through prompt engineering. However, these approaches generally adopt a problem-specific paradigm, applying a sin…

Handle-based Mesh Deformation Guided By Vision Language Model
Xingpeng Sun, Shiyang Jia, Zherong Pan, Kui Wu, Aniket Bera
https://arxiv.org/abs/2506.04562 …

Handle-based Mesh Deformation Guided By Vision Language Model
Mesh deformation is a fundamental tool in 3D content manipulation. Despite extensive prior research, existing approaches often suffer from low output quality, require significant manual tuning, or depend on data-intensive training. To address these limitations, we introduce a training-free, handle-based mesh deformation method. % Our core idea is to leverage a Vision-Language Model (VLM) to interpret and manipulate a handle-based interface through prompt engineering. We begin by applying cone s…

This https://arxiv.org/abs/2410.08911 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csSE_…

Test-driven Software Experimentation with LASSO: an LLM Prompt Benchmarking Example
Empirical software engineering faces a critical gap: the lack of standardized tools for rapid development and execution of Test-Driven Software Experiments (TDSEs) -- that is, experiments that involve the execution of software subjects and the observation and analysis of their "de facto" run-time behavior. In this paper we present a general-purpose analysis platform called LASSO that provides a minimal set of domain-specific languages and data structures to conduct TDSEs. By empowering users wi…

Designing Effective LLM-Assisted Interfaces for Curriculum Development
Abdolali Faraji, Mohammadreza Tavakoli, Mohammad Moein, Mohammadreza Molavi, G\'abor Kismih\'ok
https://arxiv.org/abs/2506.11767

Designing Effective LLM-Assisted Interfaces for Curriculum Development
Large Language Models (LLMs) have the potential to transform the way a dynamic curriculum can be delivered. However, educators face significant challenges in interacting with these models, particularly due to complex prompt engineering and usability issues, which increase workload. Additionally, inaccuracies in LLM outputs can raise issues around output quality and ethical concerns in educational content delivery. Addressing these issues requires careful oversight, best achieved through coopera…

Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements
Seyed Moein Abtahi, Akramul Azim
https://arxiv.org/abs/2506.10330

Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements
This study examined code issue detection and revision automation by integrating Large Language Models (LLMs) such as OpenAI's GPT-3.5 Turbo and GPT-4o into software development workflows. A static code analysis framework detects issues such as bugs, vulnerabilities, and code smells within a large-scale software project. Detailed information on each issue was extracted and organized to facilitate automated code revision using LLMs. An iterative prompt engineering process is applied to ensure tha…

LLM-Driven Instance-Specific Heuristic Generation and Selection
Shaofeng Zhang, Shengcai Liu, Ning Lu, Jiahao Wu, Ji Liu, Ke Tang
https://arxiv.org/abs/2506.00490

LLM-Driven Instance-Specific Heuristic Generation and Selection
Combinatorial optimization problems are widely encountered in real-world applications. Designing high-quality heuristic algorithms that efficiently approximate optimal solutions within reasonable time is a critical research challenge. In recent years, many works have explored integrating Large Language Models (LLMs) with Evolutionary Algorithms to automate heuristic algorithm design through prompt engineering. However, these approaches generally adopt a problem-specific paradigm, applying a sin…