{"id":645,"date":"2026-09-15T10:09:03","date_gmt":"2026-09-15T01:09:03","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/15\/migrating-large-prompts-to-local-ollama\/"},"modified":"2026-09-20T17:37:25","modified_gmt":"2026-09-20T08:37:25","slug":"migrating-large-prompts-to-local-ollama","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/15\/migrating-large-prompts-to-local-ollama\/","title":{"rendered":"Challenges of Migrating AI Agents to Local Ollama Environments"},"content":{"rendered":"

Overview<\/h2>\n

As developers attempt to migrate agents from commercial LLM APIs like Claude (Opus) to self-hosted environments like Ollama for privacy protection, technical challenges caused by massive pre-prompts are becoming a subject of debate. In particular, the focus is on the “thrashing” phenomenon, where agents exhibit unintended behavior due to context window limitations, as well as the nature of prompt design.<\/p>\n

Why It Is Being Discussed<\/h2>\n

This discussion was sparked when a developer shared their experience of attempting to migrate an agent using a massive 35kb pre-prompt from a commercial API to a local Ollama environment. The post gained traction on Hacker News, garnering 139 points and 76 comments, attracting the interest of many engineers.<\/p>\n

In the background, there are strong concerns regarding data collection by frontend providers and the unintended use of session metadata. The developer argues for the necessity of “Bring Your Own Weights (BYOW)”\u2014performing inference on one’s own hardware to protect ideas and highly confidential work.<\/p>\n

Discussion Points<\/h2>\n

The discussion is mainly centered around three technical points:<\/p>\n

Context Depletion and Agent Behavior<\/h3>\n

While commercial APIs offer massive context windows, local environments face strict limitations. For instance, in a system with a 65k tokens context window, a 35kb prompt immediately consumes 14% of the total. When the context saturates, a phenomenon known as “thrashing” occurs, where the agent repeats the same tool calls, re-reads already loaded files, or rewrites completed work. It is reported that this puts the agent in a state where it cannot retain immediate instructions, resembling giving instructions to “a human who reincarnates every 90 seconds.”<\/p>\n

Prompt Design Dependencies and the “Chain of Thought” Trap<\/h3>\n

It has been pointed out that the massive context windows provided by frontend providers may actually be masking poor prompt design. When a model uses Chain of Thought (CoT) for reasoning, a wide context allows the model to internally infer and compensate even for improperly constructed prompts. However, in local environments with restricted contexts, this “hidden compensation” fails, bringing design flaws in the prompt to the surface.<\/p>\n

Hardware and Tool Constraints in Local Execution<\/h3>\n

Migrating to a self-hosted environment requires immense hardware resources. Additionally, software choices are debated, with technical considerations regarding optimizing the runtime environment, such as choosing between Ollama or the lower-layer llama.cpp.<\/p>\n

Community Reactions<\/h2>\n

Various opinions are being exchanged within the community regarding prompt design philosophies and the realities of local environments.<\/p>\n

Critical Perspectives on Prompt Design<\/h3>\n

Some participants pointed out harshly that using a massive 35kb prompt itself is confusing and inefficient for any LLM. Opinions suggested that excessively long prompts reduce the effective context where the model can accurately pay attention, and thus prompts should be split by a single purpose, advocating for more declarative agent definitions.<\/p>\n

Concerns Over the Feasibility of Local Environments<\/h3>\n

Many comments addressed the high hardware requirements for operating local models. Discussions highlighted that even with expensive equipment, performance may not meet expectations, and securing practical context sizes locally remains difficult. Doubts were also raised as to whether relatively small models, such as 27b parameter models, can adequately perform advanced security tasks.<\/p>\n

Trade-offs Between Privacy and Convenience<\/h3>\n

On the other hand, many voices agreed with the distrust toward commercial providers’ data handling practices. Some opinions viewed the movement to avoid over-reliance on cloud computing and build one’s own environment as inevitable from a security standpoint. Furthermore, movements to explore concrete workarounds were shown, such as utilizing new tools to assist with context management or logging session state to disk to perform frequent handoffs.<\/p>\n

Sources<\/h2>\n