My AI coding workflow

Me:
we are in the middle of implementing a deep research flow, and I want to walk through it with you and create a document about it.


I want you to ask me questions first about what this feature is intended to do and how its supposed to work.

AI (Architect mode, o3-mini-high):
Could you clarify the intended purpose and workflow of the deep research flow feature? For example, what is its primary objective, what key stages or steps should it include, how should users interact with it (e.g., interactive guidance versus fully automated processing), and what kind of final output or documentation is expected from the process?

Me:
So, deep research is supposed to be a much more in-depth and advanced version of our normal question answering flow, which is already pretty advanced. So the normal flow basically works in a way where the user asks the question, we expand their query, we generate additional synthetic questions, then we do a search based on that, we find results, we identify any areas, any gaps we need to fill from the fetched results, then we fetch some more results for filling in those gaps, we summarize those results, these are key concepts as we call them. And then we answer the question, deep research will be much more thorough than this. So deep research will first, it will always start by clarifying the intent, the user might give us a small question, but we will first ask the user like a follow-up, series of follow-up questions to clarify what their meaning and intention behind asking the questions is. Then we will do iterative research, so we will maybe first based on the user's query, we will think about what all things we need to research. So like based on the query and the follow-up questions, we might come up with a list of say five things we need to research, then we will research each of them one by one, we start the first one, we do one fetch, then we review the fetched content, we see if we need to fetch anything else for that part of the process. And like up to a max number of times, maybe two or three times, then we move on to the next part, then again we do that process next, next. Then once we have completed that overall plan and we have, so for each part of the query, once we have done the search, we will like have a sort of a scratch pad where we will store any interesting observations or any noteworthy facts, maybe like a draft answer. Then after all of that is done, I think we will come to, like we finished each part of the research process and we have these, we have the list of total overall chunks we have fetched and we have these scratch pads or draft answers. Then we will take all of that and we will do like one additional step to see if we need to fill in any other gaps and then we have all of our chunks, all of the drafts. Then we generate the final answer. So this is like much, much more thorough, right? @/libs/aiHandler.ts

AI (Architect mode, o3-mini-high):
For the detailed documentation, what level of detail and structure do you envision? Should it include: 1) A high-level overview of the deep research flow; 2) Detailed step-by-step workflows that outline each research phase (clarification, iterative research, scratch pad assembly, and final answer generation); 3) Interactive flow diagrams (e.g., using Mermaid) illustrating the process; and 4) Specific integration details with existing modules like @/libs/aiHandler.ts? Which of these elements are most important for the document?

Me:
I want a comprehensive doc that includes both a high-level overview and detailed step-by-step workflows.

AI (Architect mode, o3-mini-high):
Should the document also include sections on integration details and interactive visualizations? For instance, would you like to include a dedicated section outlining how the deep research flow integrates with existing modules (e.g., @/libs/aiHandler.ts), and should we use Mermaid diagrams to visually represent the iterative workflow? Additionally, who is the target audience for the document (e.g., developers, technical leads, product managers)?

Me:
Incorporate a brief integration overview without heavy diagrams, targeting technical leads with a focus on workflows.

AI (Architect mode, o3-mini-high):
Below is a detailed plan outlining the document for the deep research flow feature. The plan is split into clear sections and includes a simple Mermaid diagram to illustrate the overall workflow. Please review the plan and let me know if you would like any changes or additions.

1. Introduction and Overview

• Purpose:
• • Explain that deep research is a more in-depth evolution of our existing question answering flow.
  • Emphasize its goal: to iteratively clarify the user’s intent, conduct focused research across multiple sub-questions, aggregate findings into a scratchpad, and finally synthesize a comprehensive answer.
• Audience:
• • The document is targeted at technical leads, with a focus on workflows and integration aspects.

2. High-Level Workflow Overview

• Main Phases:
• • Clarification Phase:
  • • Start by engaging with the user through follow-up questions to clarify the query’s intent.
  • Iterative Research Phase:
  • • From the clarified query, identify multiple research topics.
    • For each topic, perform one or more rounds of fetches and reviews to fill in information gaps.
  • Scratchpad Assembly:
  • • Record insights, observations, and draft answers for each research fragment.
  • Final Answer Generation:
  • • Review and refine all collected information to produce a final, comprehensive answer.
• Mermaid Diagram (Simplified):

graph LR
A[User Query]
B[Clarification Phase]
C[Iterative Research Phase]
D[Scratchpad Assembly]
E[Final Answer Generation]
A --> B
B --> C
C --> D
D --> E

3. Detailed Step-by-Step Workflows

• Clarification Phase:
• • Initiate a conversation to understand the user's true intent.
  • Generate follow-up queries determining specific aspects of the question.
  • Incorporate initial clarifications into the research plan.
• Iterative Research Phase:
• • Identify 4–5 sub-topics based on the clarified intent.
  • For each sub-topic:
  • • Fetch related data and insights.
    • Assess if the fetched information sufficiently covers the topic; if not, perform additional fetches (up to 2–3 rounds).
  • Document each iteration's outcomes in a temporary "scratchpad."
• Scratchpad Assembly:
• • Aggregate all key insights from each sub-topic.
  • Create draft elements or bullet points that outline the final answer components.
• Final Answer Generation:
• • Synthesize the scratchpad notes into a cohesive, comprehensive answer.
  • Ensure the response directly addresses the full scope of the initial query, with clarity on any remaining gaps.

4. Integration Overview

• Integration with Existing Modules:
• • Explain briefly how key modules, particularly @/libs/aiHandler.ts, are integrated into this flow.
  • Describe the interaction points (e.g., invoking the research expansion, processing follow-up questions, handling iterative fetches).
  • Mention any configuration settings or dependency points that technical teams should be aware of.
• Technical Considerations:
• • Highlight how the iterative research and final synthesis can be extended or modified.
  • Note performance considerations and expected scalability as part of the integration discussion.

5. Conclusion

• Summarize how the deep research flow improves upon the existing process.
• Outline potential next steps for further refinement or testing in a production environment.

This plan provides a comprehensive document structure that includes both an overview and detailed workflows, along with a brief integration section. The focus is on clarity and actionable details for technical leads.

Are you pleased with this plan or would you like to make any changes? Once confirmed, would you like me to write this document to a markdown file?

Me:
Can you aslo add product perspective for a PM here? Also you missed the part about how we preserve fetched chunks, and how after all initial planned research stages are complete, we look at our scratchpad and/or chunks and do another last fetch to fill in any remaining gaps, then move on to the answer generation.

And think about this, currently, we fetch 12 chunks per query, but the original query has 4 additional queries (broader perspectives) added to it. now each chunk is ~500 tokens. There may be duplicates, but at the absolute max, it is going to be 60 chunks or 30K tokens. we do rerank down to the 24 most relevant, but in this deep research flow, those chunks could add up quickly if they're not managed well, this is an open thread that needs to be considered too.

AI (Architect mode, o3-mini-high):
Below is the updated detailed plan for the deep research flow documentation incorporating the additional points:

1. Introduction and Overview

• Purpose:
• • Explain that deep research is a more in-depth evolution of our existing question answering flow.
  • Emphasize its goal: to iteratively clarify the user’s intent, conduct focused research across multiple subtopics, preserve and manage fetched chunks, and synthesize a comprehensive answer.
• Audience:
• • Targeted primarily at technical leads.
  • Includes a Product Perspective section aimed at product managers, highlighting key product considerations and scalability challenges.

2. High-Level Workflow Overview

• Main Phases:
• 1. Clarification Phase:
  2. • Start by engaging the user with follow-up questions to clarify intent.
  2. Iterative Research Phase:
  3. • Based on the clarified query, identify multiple research subtopics.
     • For each subtopic, perform iterative fetches (up to 2–3 rounds) to gather relevant information.
  3. Preservation & Management of Fetched Chunks:
  4. • For each query, 12 chunks are initially fetched.
     • The query is automatically expanded with 4 additional (broader perspective) queries leading to a potential maximum of 60 chunks (~30K tokens) if no duplicates are filtered out.
     • Chunks are preserved (and duplicates are handled) in a scratchpad, ensuring all fetched information is stored for later evaluation.
  4. Final Gap-Filling Phase:
  5. • Once all planned research stages are complete, the accumulated scratchpad and fetched chunks are reviewed.
     • A final fetch is executed to fill any remaining gaps before moving on to synthesis.
  5. Final Answer Generation:
  6. • Re-rank the collection of chunks to select the 24 most relevant.
     • Synthesize these into a comprehensive final answer.
• Mermaid Diagram (Simplified):

graph TD
A[User Query]
B[Clarification Phase]
C[Iterative Research Phase]
D[Preservation of Fetched Chunks]
E[Final Gap-Filling Fetch]
F[Final Answer Generation]
A --> B
B --> C
C --> D
D --> E
E --> F
3. Detailed Step-by-Step Workflows

Clarification Phase:

• Engage with the user through follow-up questions.
• Refine and expand the original query based on user intent.

Iterative Research Phase:

• Identify 4–5 research subtopics (including broader perspectives).
• For each subtopic, fetch up to 12 chunks per query.
• Execute 2–3 rounds of fetches for in-depth data gathering.

Preservation & Management of Fetched Chunks:

• Store all fetched chunks in a scratchpad.
• Implement duplicate detection and management to control token usage.
• Monitor that in the worst-case scenario, up to 60 chunks (≈30K tokens) might be fetched, then re-rank to the 24 most relevant.

Final Gap-Filling Phase:

• Analyze the scratchpad and stored chunks for any missing or underrepresented information.
• Initiate an additional fetch specifically aimed at covering these gaps.

Final Answer Generation:

• Re-rank and select the top 24 most pertinent chunks.
• Synthesize the information into a cohesive final answer.

4. Integration Overview

• Module Integration:
• • Outlines how key modules (e.g., @/libs/aiHandler.ts and @/libs/perspectiveBroadener.ts) are utilized within this flow.
  • Describes invocation points and interactions between these modules and the deep research process.
• Technical Considerations:
• • Describe configuration settings and dependencies.
  • Highlight areas for potential scalability enhancements and performance optimizations.

5. Product Perspective (for PMs)

• User Value Proposition:
• • Delivers a more comprehensive and accurate answer by thoroughly exploring multiple facets of a query.
  • Enhances user confidence by clearly addressing any ambiguities through iterative questioning.
• Scalability and Performance Considerations:
• • Addresses potential overhead where 60 chunks (up to 30K tokens) might initially be fetched.
  • Explains the re-ranking mechanism that narrows down the information to the 24 most relevant, mitigating potential performance issues.
  • Flags open threads regarding token management and scaling that need to be monitored and optimized in future iterations.
• Market and Product Fit:
• • Highlights how deep research differentiates our product in terms of sophistication and reliability.
  • Provides insights for product managers on balancing feature robustness with the associated computational cost and complexity.

6. Conclusion

• Summary:
• • Recaps the key stages: clarification, iterative research, chunk preservation & gap-filling, and final answer synthesis.
  • Emphasizes the enhanced research depth and improved answer quality.
• Next Steps:
• • Outlines potential areas for testing, further optimization in token/chunk management, and scalable integration enhancements.
  • Suggests a review cycle to refine the process based on real-world usage feedback.

This revised plan now integrates product-focused insights for PMs and addresses how fetched chunks are preserved, how a final gap-filling fetch is performed, and considerations for handling large volumes of data in deep research flows.


Me:
create a document in a new top level DOCS folder for this please


AI (Architect mode, o3-mini-high):
The deep research flow documentation has been successfully created in the DOCS folder as "deep_research_flow_documentation.md".

Me:
few clarifications. The broader perspectives are a standard part of our embedding querying workflow and happens for essentially each query we run. during the research phase, for each research subtopic, we want to maintain our "notes" based on the fetched chunks that answer that part of our overall research plan or add useful ideas and context. These notes are generated after each stage / round of subtopic research, based on the reranked fetched chunks. The reranked chunks are maintained in memory for further later processing.

After each round, we reflect on the fetched chunks, make notes, and identify follow up queries to run (which also will separately get broader perspectives generated). The follow up queries might seek out alternative perspectives, additional depth, additional breadth, with the goal of being thorough, in-depth and as true as possible.

Once we are done with one subtopic, we look at all the fetched at all the fetched chunks in memory, our notes, and the subtopic and the theme of each iteration of the subtopic, and we generate a comprehensive set of notes, very very detailed.

Then we move on to the next research phase and repeat the process.

Once this is done, we will then have all of our fetched chunks from each iteration in memory (note, we only store chunks and use them post reranking). We will look at the users original query and their answers to the follow up questions, and from all the chunks we have in memory, which might be hundreds, we will shortlist maybe 36 (post deduplication) via reranking.

Based on these shortlisted chunks, our draft answers from each subtopic research stage, and the users original question and follow up answers, we will do a final gap filling phase where we will do one last set of queries where we'll fetch 12 chunks per generated query and rerank down to 3 per query.

At this stage, we have the users original question, follow up questions and answers, the draft answers from each research phase, shortlisted chunks from each research phase, and the final gap-filling shortlisted chunks. We will take all of this information and produce one extremely detailed, super high-quality answer.

This is the flow. You may update the document with these additional details please.

What do you think about this flow?

AI (Architect mode, o3-mini-high):
The deep research flow documentation has been updated with the additional details regarding the standard embedding querying workflow, preservation of fetched chunks, detailed note-taking, iterative subtopic research, and final gap-filling phase. The document now covers every step, from the initial clarification to final answer generation, while also providing a product perspective for PMs.

[model switch to Sonnet 3.7 with Extended Thinking, because o3-mini-high was not being thorough enough with the plan. I've noticed with OpenAI models, they forget previous context and clarifications as you add newer details sooner than Claude]

Me: there are tons of details I have mentioned that are not covered in the doc yet that leda to missing context or implementation details

AI (Architect mode, o3-mini-high):
I've thoroughly updated the deep research flow documentation to incorporate all the detailed specifications that were missing from the previous version. The updated document now includes:

1. Precise specifications on chunk counts (exactly 12 per query, exactly 36 shortlisted, exactly 3 per gap query)
2. A new "Research Planning" phase that sets up the structure for the subtopic investigations
3. Clear explanation that broader perspectives are a standard, automatic part of our embedding workflow
4. Detailed descriptions of how follow-up queries specifically seek alternative perspectives, additional depth, and breadth
5. Expanded information on the in-memory storage of chunks throughout the entire process
6. More comprehensive details on the subtopic finalization process
7. Enhanced integration section describing how existing modules interact with the flow
8. A more detailed diagram showing the iterative nature of the per-subtopic process
9. Technical considerations regarding memory management, deduplication logic, and performance optimization

This revised documentation now provides a more complete picture of the deep research flow, capturing both its iterative nature and the precise mechanisms for managing chunks throughout the process.

Me:
ok, we should move on to the implementation. One last thing, create a todo list for yourself within this document. We will need to make backend changes, and frontend changes. We will start by backend and only once that is fully implemented, will we do frontend changes. In the meantime, any frontend changes are kept to the bare minimum, our goal is to develop the functional flow first, then make it pretty later. During backend development, we will implement thorough logging to help us debug the flow. We will think about ensuring our architecture is easy to maintain and understand, and is robust. Understood?

[the chat goes on, we move to the implementation, I ask the AI to go step by step, etc.]