We Migrated from Assistants API to Responses API: What Changed in Production

Our AI system faced major performance issues - delays, frozen threads, and inefficient state management. With the Assistants API set to shut down in August 2026, we switched to the Responses API early. Here’s what improved:

• Response times dropped by 60%. Polling delays are gone, and responses now stream instantly.
• Costs fell by 40–60%. Better token efficiency and caching cut expenses significantly.
• Simpler architecture. previous_response_id replaced manual state tracking, reducing code complexity.
• Improved tool integration. Multiple tools can now run within a single request, improving reliability.

The migration wasn’t without challenges - manual thread migrations, syncing vector stores, and adapting to new pricing were key hurdles. But the move positioned us to handle more traffic, reduce costs, and deliver faster, smoother customer experiences.

Key takeaway: The Responses API isn’t just faster - it simplifies workflows and enables better AI capabilities for the future.

Our System Before the Migration

CoSupport AI relied on the Assistants API for over a year to manage its production environment, handling thousands of customer support conversations daily. The architecture was built around four core components: Assistants (configuration), Threads (session state), Messages (content), and Runs (execution). While this setup got the job done initially, scaling up revealed some serious flaws, including operational inefficiencies and performance bottlenecks.

Key takeaway: Early monitoring of production metrics can help spot scaling issues before they spiral out of control.

How We Used the Assistants API

We designed separate Assistant objects for different workflows, such as technical troubleshooting, billing inquiries, and general questions. Each customer interaction spun up a new Thread to store the conversation history server-side, while Runs handled the actual execution of tasks.

The system's asynchronous execution model added complexity. After starting a Run, our system had to repeatedly poll its status, leading to delays of 5–10 seconds before users saw the first response.

Thread locking made things worse. While a Run was active, the entire Thread was locked, preventing any new interactions. If a customer sent a follow-up message during this time, it would queue until the Run finished. During busy periods, response times could balloon to 10–20 seconds, frustrating both users and support teams.

Tool integrations were tightly coupled with Assistant objects, meaning even minor updates - like tweaking File Search behavior or adjusting Code Interpreter settings - required programmatic changes. This made A/B testing nearly impossible and forced full code deployments for what should have been simple configuration updates.

"Version management was cumbersome due to its programmatic definition." - Murtuzaali Surti, AI Developer

Image reference: A diagram illustrating the Assistants API flow: Assistant → Thread → Messages → Run → Run Steps → Polling Loop → Response

These structural limitations eventually led to more severe production challenges.

Main Problems We Faced

Latency was the biggest issue. Response streaming typically took 5–10 seconds to start, with most requests taking 10–20 seconds to complete. During high-traffic times, some Runs would get stuck in an indefinite "running" state, requiring manual intervention to cancel them. In some cases, cancellations would hang, forcing us to abandon the Thread entirely and restart the conversation.

State management became a major headache. Threads only stored messages, while tool calls and outputs were saved separately as Run Steps. This fragmented approach made debugging incredibly difficult, as piecing together the full reasoning chain required pulling data from multiple API objects. Understanding why the AI made certain decisions - or identifying the source of errors - was nearly impossible.

Observability was another weak point. The generalized nature of Run Steps didn’t align well with our monitoring tools, making it hard to track essential metrics like tool call success rates, failure rates, or execution times. Issues with vector store synchronization further complicated matters; inconsistencies often came to light only after customers reported incorrect answers.

Token efficiency also suffered. The architecture didn’t support effective prompt caching, leading to poor cache utilization. Internal evaluations later revealed that newer systems could achieve 40%–80% better caching, but we were stuck with the legacy design. Combined with the overhead of asynchronous processing, our API costs remained stubbornly high despite optimization efforts.

"The vector stores are flakey. Uploading files works but we maintain a database of what we have uploaded and often find that when we list files via API, we find that they are out of sync and have to delete and upload to correct this." - kduffie, Chief AI Architect, NeuroGen Labs

These challenges made it clear: our system needed a major overhaul. The operational inefficiencies and performance issues pushed us to explore a more streamlined alternative with the Responses API.

Key takeaway: A functional API isn’t enough for production systems. To scale effectively, you need predictable performance, robust observability, and reliable state management.

sbb-itb-97114f1

How We Migrated from Assistants API to Responses API

Switching from the Assistants API to the Responses API was a complex process that required careful planning to avoid interrupting production. We divided the migration into three distinct phases: planning, building and testing, and production rollout. Each phase addressed specific technical hurdles while ensuring minimal downtime for our users.

Key takeaway: Breaking the migration into phases and running systems in parallel helps reduce risks and allows for thorough validation before full deployment.

Planning the Migration

To streamline the transition, we mapped components from the Assistants API to their equivalents in the Responses API: Assistants became Prompts, Threads turned into Conversations, Runs were converted to Responses, and Run Steps were replaced with Items. This mapping not only simplified the migration but also resolved longstanding issues like polling delays and inefficient state management.

We replaced server-side state management with a previous_response_id system that allowed our application code to handle orchestration tasks such as history pruning, tool loops, and retries. This shift enabled prompts to focus solely on high-level behavior and constraints.

Instead of migrating everything in one go, we prioritized high-traffic workflows that would benefit most from the Responses API's enhanced reasoning capabilities. Legacy flows remained on the Assistants API temporarily to ensure stability during the transition.

Image reference: A comparison table showcasing the mapping between Assistants API objects (Assistants, Threads, Runs, Run Steps) and their Responses API counterparts (Prompts, Conversations, Responses, Items).

We also updated function configurations to align with the Responses API's stricter mode and polymorphism. To maintain consistency throughout the rollout, we stored prompt IDs in source control for stable identifiers.

Our phased approach included:

• Gradually introducing the Responses API for new user chats while keeping ongoing conversations on the legacy API.
• Testing improvements on high-traffic workflows before expanding the migration.
• Maintaining a rollback plan to address any unexpected issues.

With the architecture defined, we moved on to building and testing the system.

Building and Testing

The old system’s limitations - such as 10–20 second delays and fragmented state management - shaped our technical decisions during development. We refactored the orchestration code in TypeScript, leveraging OpenAI SDK v4’s type definitions to cut down debugging time.

"The transition to the Responses API took only about 2 hours of work... since we use TypeScript, the typedefs helped immensely in making this process smooth." - kduffie, Developer

Running both APIs in parallel was essential for validation. We compared outputs from the old and new systems to ensure feature parity before directing production traffic to the new API. Using the dashboard’s prompt versioning feature, we could snapshot and compare configuration specs without altering application code.

Custom hooks were built to handle Server-Sent Events, and we implemented an AsyncTimedIterable wrapper to manage stream timeouts for long-running reasoning tasks. We also focused on error handling, adding specific handlers for response.failed and error events to minimize disruptions for users.

To maintain backward compatibility, we created scripts to backfill old Assistant threads into new Conversation objects, ensuring that existing users retained their context.

After confirming the system’s performance in controlled tests, we prepared for a gradual rollout in production.

Going Live in Production

The production rollout began incrementally, starting with new user chats while retaining existing conversations on the Assistants API. We closely monitored key metrics like latency, token usage, and tool call success. The Responses API’s improved observability made tracking these metrics much easier compared to the old system.

One developer at a chat application company supporting tens of thousands of users completed a similar migration in just two hours using TypeScript in March 2025.

Throughout the initial deployment, we maintained a rollback plan. If any critical issues arose, we could quickly revert traffic to the Assistants API while troubleshooting. However, running both systems in parallel during testing caught most issues, making the production rollout relatively smooth.

The final phase involved migrating the remaining legacy threads. Active conversations were prioritized, followed by historical data, which was backfilled over several days. By the end of the process, all new interactions utilized the Responses API, while existing users experienced a seamless transition.

Key takeaway: Gradual rollouts with parallel testing and strong monitoring ensure smooth migrations. Using tools like strong typing and dashboard-based prompt management significantly reduces development time and risk during transitions.

What Improved After the Migration

With the migration from the Assistants API to the Responses API complete, the results speak for themselves. Within just a few weeks, the benefits became clear, with measurable improvements that impacted both the production environment and the experience for end users.

Key takeaway: The Responses API brought faster response times, lower costs, and a smoother customer experience, thanks to its modern architecture.

Performance Results

Response latency saw a dramatic 60% reduction, primarily because the new system eliminated polling delays. Previously, the old setup required multiple API pings to check if a task was finished, adding an extra 10–20 seconds to interactions. The Responses API, with its native streaming, provided instant, word-by-word responses, solving the fragmented state management issues we had encountered.

Token management also became far more efficient, improving by 40% to 80% compared to the Chat Completions API. This was especially valuable during multi-turn conversations, where maintaining context awareness is essential for a smooth user experience.

Scalability improved significantly, thanks to the simplified architecture. By replacing server-side Threads with the previous_response_id system, we cut down on orchestration complexity and removed bottlenecks that had previously limited our ability to handle concurrent requests. This allowed us to support higher traffic levels without needing major infrastructure upgrades.

Image reference: A line graph showing a 60% reduction in average response latency (in seconds) over 30 days, comparing the Assistants API and Responses API.

Tool call reliability also improved. The Responses API's explicit tool loop handling gave us greater control over function execution, reducing failures and timeouts that had occasionally disrupted the legacy system. These performance gains not only improved user experience but also led to notable cost savings.

Cost Savings

API costs dropped by 40% to 60% within the first month of deployment. This was mainly due to better cache utilization, which reduced redundant token processing and directly lowered our monthly OpenAI expenses. For a system managing thousands of daily support conversations, these savings were substantial.

"The results exceeded our expectations: 60% faster responses, 40-60% cost reduction, and dramatically simplified maintenance." - Jürg Steudler, Senior Salesforce Developer

New tools called charges, such as $2.50 per 1,000 file searches, encouraged us to optimize workflows for cost efficiency. For workflows heavily dependent on retrieval-augmented generation, this required careful monitoring and adjustments.

Infrastructure expenses also went down. The simpler orchestration logic reduced server-side processing demands, enabling us to scale down backend resources. Additionally, the cleaner API design meant fewer edge cases and less debugging, cutting maintenance overhead. These cost reductions, combined with faster and more accurate responses, significantly enhanced the customer support experience.

Better Support Experience

Streaming responses completely changed how customers interacted with the system. Gone were the days of waiting 15–20 seconds for a full reply. Now, users see responses start within 5–10 seconds, with most conversations wrapping up in under 20 seconds. This created a natural, conversational flow and cut down on frustrating wait times.

Context retention also became more reliable. The previous_response_id The system kept conversations flowing smoothly, avoiding the fragmentation issues that sometimes occurred with the old Thread-based management. Customers reported fewer instances of the AI "forgetting" earlier parts of the discussion, leading to more coherent multi-turn interactions.

Access to advanced reasoning models via the Responses API further improved accuracy. OpenAI's internal evaluations showed a 3% boost in SWE-bench scores using identical prompts. This meant that even complex technical questions were answered with greater precision and depth compared to the previous system.

Key takeaway: While the performance and cost improvements are impressive, the real win is a faster, more reliable customer support experience that users genuinely appreciate.

What We Learned and What's Next

Our migration highlighted some tough challenges with state management and keeping data consistent. One major issue was that the Responses API doesn’t automatically carry over system prompts between requests. To address this, we updated our application logic to include the same instructions string with every call. This ensured the AI maintained its persona throughout conversations.

We also had to manually handle thread migration. OpenAI doesn’t provide an automated way to convert existing Threads into the new Conversations model. To adapt, we manually backfilled old conversations to turn threads into new Items. For teams with large historical datasets, having a clear plan for archiving is essential to avoid losing data.

Another hurdle was syncing our vector store. Occasionally, files would fall out of sync between the API and our local database. To mitigate this, we implemented regular health checks and automated scripts to reconcile any discrepancies. These proactive measures were key to keeping production stable.

Key takeaway: Backward compatibility can be tricky, but thorough testing, smart state management, and constant monitoring make it manageable.

Problems We Encountered

The transition forced us to rethink how we managed the state. While the previous_response_id system made context management easier, it only retains objects for 30 days. To preserve long-term conversation history, we switched to the conversation_id model and developed our own archival solution.

The move to dashboard-managed prompts also required us to adjust our workflows. Unlike Assistant objects, Prompts can’t be created programmatically and must be versioned through the OpenAI dashboard. This change reduced some of the automation we relied on for fast testing, requiring manual intervention instead.

"The transition to the Responses API took only about 2 hours of work. The structure of the API is completely different, but it's simple and straightforward." - kduffie, Developer

Another challenge came from the new pricing structure. The $2.50 per 1,000 file search calls significantly increased costs for workflows that rely heavily on retrieval. To manage this, we optimized how often we triggered file searches and introduced caching to cut costs without compromising response quality.

Key takeaway: Migrating involves more than just code changes. Be ready for workflow adjustments, new cost considerations, and a temporary dip in automation.

Best Practices from Our Experience

To navigate these challenges, we developed a set of best practices. Running both APIs in parallel during the migration was a lifesaver. By routing a small percentage of traffic to the new system while keeping the old one as a fallback, we could catch issues early and roll back quickly if needed.

TypeScript proved to be incredibly helpful. Its strong type definitions caught structural changes during refactoring, saving us from runtime errors that could’ve been a nightmare to debug. If you’re not using TypeScript yet, this is a great opportunity to start.

Token monitoring became a priority after switching to previous_response_id. With all prior messages counting as input tokens for billing, long conversations can quickly rack up costs. We set up automated alerts for conversations breaching token limits and used the truncation: "auto" setting to automatically discard older messages.

Testing newer models, like GPT-5 and o4-mini, revealed they behave differently from GPT-4. This meant we had to tweak our prompts and validation logic to align with the new models. Don’t assume your existing prompts will work seamlessly with updated models.

Key takeaway: Gradual rollouts, strong typing, and vigilant monitoring are crucial for a smooth migration.

Our Plans for the Responses API

The migration didn’t just improve performance - it opened up opportunities for exciting new capabilities. Next, we’re focusing on implementing the Model Context Protocol (MCP). This will let us connect AI agents to external data sources using a standardized protocol, cutting down on the need for custom integrations and making it simpler to add new tools.

We’re also expanding how we use vector store metadata. Adding attributes like product category, date range, and content type will allow for more precise searches, improving response relevance and ensuring hallucination-free AI in specialized situations.

Another area of exploration is multi-tool loops. By combining web search, file search, and code interpretation into a single request, we’ve started handling complex queries more efficiently. Early tests suggest this approach reduces latency and enhances the user experience, especially for technical troubleshooting.

Finally, we’re working on building a provider-agnostic architecture using the open-source Agents SDK. This setup will let us easily switch between OpenAI, local models, or other providers with minimal code changes. It’s a step toward greater flexibility as the AI landscape continues to evolve.

Key takeaway: The Responses API isn’t just a replacement - it’s a foundation for creating advanced AI agents with better tools, cleaner workflows, and room to grow.

How the Migration Improved Our System

Switching to the Responses API brought noticeable improvements: better performance, lower costs, and a smoother workflow for our developers. By simplifying state management, we eliminated the hassle of manual thread tracking and fixed issues caused by fragmented state handling. Operations became more efficient thanks to streamlined tool calls, and cache usage soared by 40% to 80% in production, directly cutting operational expenses.

The new conversation-based model transformed how we handle context. Instead of juggling multiple objects, we now simply use a previous_response_id or a conversation_id to maintain the flow. This reduced code complexity, making debugging far easier - especially with TypeScript's strict typing catching potential errors early.

Performance metrics also told a clear story. Internal testing showed a 3% boost in SWE-bench scores when reasoning models used the Responses API instead of Chat Completions. For CoSupport AI, this meant quicker response times and more precise support interactions, which translated into a better overall experience for users. These improvements pave the way for future scaling and the addition of new features.

"The Responses API represents the future direction for building agents on OpenAI." - OpenAI

Beyond the immediate benefits, this migration positions us well for what’s ahead. With the Assistants API set to sunset on August 26, 2026, moving early gave us access to advanced features like deep research capabilities and sophisticated reasoning models. While the dashboard-managed Prompts system required some workflow updates, it introduced better version control and made testing behavior changes easier without touching application code.

The bottom line: The Responses API isn’t just a replacement - it’s a robust platform that drives scalability while delivering real improvements in speed, cost efficiency, and developer productivity.

See how it works in practice. Get a free demo.