← portfolioMadam, OTP? · Project Kabootar

09 · Final Round Whimsical: Transcription

info · Transcribed from your two large Whimsical boards: "delivery agent simulations brainstorming and changes to make for final version" (Board 1) and the big questions / madam otp / normal flow / return / agent-human flow / stages board (Board 2). Text is kept as written. […] marks a place where the board image is cut off at a seam or the text was too small to read with confidence; those spots are listed in 00 README Index so you can fill them from the live Whimsical.


BOARD 1 · "delivery agent simulations brainstorming and changes to make for final version"

1.1 Round 3 brief (pasted on the board, verbatim)

n Round 2 you designed your agent on paper. This round has two parts. First you run a simulation of your agent. Then you work out which company turns it into a business, and how. You'll build the simulation, using a method called Wizard of Oz. The parts that can run, run. The parts that don't exist yet are played by someone behind a curtain. Your agent is the part that runs. It's an AI model like Claude or Gemini, following a system prompt you write, and it makes every decision on its own. Wherever the model can connect to something directly, like Gmail, WhatsApp or Google Sheets, connect it. Everything else is played by a team member behind the curtain:

What your agent connects to - Gnani: run the audio or text through Gnani's API yourself, and paste back exactly what Gnani returns.. - Pine Labs and Delhivery: reply with the response their documentation says they'd send. - Anything else from the outside world, like a voice note arriving or the clock hitting 5 PM: a team member feeds it in. It must come from a real source. If no real source produces it today, your agent can't have it. The person behind the curtain never helps the agent think. They only play the world around it. If your agent needs someone to tell it what to do next, it isn't an agent yet.

Recording your simulation - Record your screen while your agent runs from the first thing that happens to the moment your agent's outcome is achieved. - The recording must show every input arriving, every call your agent makes to Gnani, Pine Labs or Delhivery and the response that comes back, and every message your agent sends to a person. - Keep it to 5 minutes at most. Fast-forward the parts where the model is thinking. - Share the link, along with your system prompt and the name of the model you used.

Part 1 (mind map, verbatim)

Part 2 (mind map, verbatim)

1.2 Simulation screens and the changes you wanted for the final version

These are the screens from the simulation tool with your handwritten change notes next to each. Screens are described only so your notes make sense; the notes are verbatim.

# Screen (as shown) Your note, verbatim
1 Driver console card "Call the recipient / Keep driving. Divya Menon / Recipient is present · Not present" add "0/3 Calls made" to track how many calls were being made
2 WhatsApp thread with Divya Menon (buyer): agent asks for dad's name and number; user replies "give it to my dad", "9898980000", "change the number", "remove dad , add 9898989898", "add this 9898989898" no confirmation of the alternate number
2 (same screen) i want to also add the not available option on whatsapp with teh leave it at door part
2 (same screen) the agent should ask to give alternative number in the format of the delivery recipient's actual name (example: Ravi Kishan) and his phone number
3 Driver app stop list with "Divya Menon (buyer) · Kettle · order OD4415 · 12:23:52 · No answer · rang out, nobody picked up" when the recipient is not there and they did not pick the voice agent call and the recipient divya sharma did not pick the call multiple times ,expected output should have been thet the call should have placed to the alternative number she mentioned , but what happened is that it tells the driver to do a manual call ,
4 Stop list with "Alternate Recipient · QR image" and "Dad · QR image" at the bottom the alternate number should be defined properly in the database and put it on top and not send it to the bottom
5 Call transcript: agent "Hi Divya, I am calling from delivery. We have a kettle for you arriving right now. Will you be available to receive it?" / Divya "i am not available leave at the door" / agent "I understand, but we can't leave it unattended at the door for safety reasons. Can someone else receive it for you?" leave at the door should be added when i am asking kabootar to leave it at the door , it responds with this , and instead it should tell okay and send a confirmation of leaving it at the door with a picture
6 Driver card "Stop 1 · Divya Menon · Kettle · 7*** , RMZ Ecoworld Residency · QR sent to Mom (alternate) · Safe to leave" "safe to leave" to order completed

1.3 WhatsApp reference screenshots

1.4 Flipkart mind map ("why flipkart") and sources

1.5 How will you monetize this agent (mind map)

1.6 Where does your agent live, and how does a customer start using it?

1.7 How will the agent get its first 1,000 users?


BOARD 2 · Questions, "madam otp", normal flow, return, agent-human flow, stages

2.1 Round 2 questions with your answers on the map (the "questions" tree)

Pasted brief (verbatim): "Design an agent that is specified clearly enough for someone else to build. The submission should clearly define: 1. What the agent is accountable for. 2. How autonomous the agent is. 3. How the agent behaves across different states and scenarios. 4. What existing capabilities the agent can use. 5. What capabilities are missing and need to be built. 6. How humans interact with the agent. 7. What the agent is called. 8. Which Indian company could potentially build an agent like this."

8. Which Indian company has the best chance of creating an agent just like yours? → flipkart and amazon · they own everything from sku to in house delivery service

1. Agent Outcome — What is the outcome your agent is accountable for? - voice agent → outcome → Understanding the context of the end user: whether he is available or not available · whether his family members are picking up · whether he wants to reschedule itand then giving that information to the delivery agent → to give a live status check if the rider s[…] that address or move to the next addre[ss] - logistic agent → delivery → outcome → removing the friction caused by OTP · returns → outcome - payment → outcome → Automatic payment for COD when the buyer is not available

2. Level of Autonomy — L1 Prepares things for the person. They still do the work. · L2 Proposes a specific action every time. The person says yes or no to each one. · L3 Acts on its own inside limits somebody set. The person deals with whatever falls outside those limits. · L4 Plans several steps, does them, checks its own work, comes back when stuck, and is judged on whether the thing got done.

3. Agent States and Flows — happy flow (cell connectivity? loud noise?) · unhappy flow (delivery agents camera is broken · end users phone is cracked)

4. Existing Rails and Missing Capabilities → can we have a call id display (delhivery/ delivery agent)

5. Fourth Rail → "what is a solution that makes it an innovation when compared to the existing technology ??" - 1) assuming otp is the problem and cause of friction - attach a camera to the delivery guy to remove the otp in intself , so assuming we can run small language models on edge devices , the computer vision software will be able to run and extract features real time and analyze which parcel is sent to whome without the otp - as the otp here solves the authentication problem which is more authentic than giving it to the same guy that ordered the delivery ? - 2)anytime anywhere and dynamic delivery application that analyzes your location real time and recommends if the delivery must be done to a specific location you are in or not - 3)telecommunication spam detection ? - 4)consumer ai which understands the users preferences and location and choices , for better machine to machine translation - (also on board) "this is why the 4th rail is the consumer ai / personal assistant"

6. Human Interface — (the seven prompts: how the user initiates interaction · how the agent communicates with the user · what information the user sees · when the agent asks for approval · when the agent acts independently · how the user handles exceptions or escalations · what the overall interaction looks like from beginning to end)

7. Agent Name — "Provide the name of the agent."

2.2 "questions to clarify" (your list, with the answers you reached)

Question Answer / note on the board
why isn't there live tracking of DA like how food delivery platform show live tracking of DA for ETA a delivery guy does not pick up and deliver one order at a time so its not a pick up to direct destination problem , it is more of a multiple pickups and multiple delivery , so when the guy is delivering first order , the first order address might be closer to my house and the second order address might be way farther , and third might be mine , so withi the live tracking info , nothing is solved and proper eta can not be calculated → also privacy issue.
how is NDR logged in general? is there any place to give reasons by the delivery agent for NDR? Ndr needs actual reasons · And it is apparently cross referenced to the call logs and location logs · If the guy did not spend enough time at a location or did not place a call at the location and marked in the ndr it is flagged as fraud
Let us say there are 15 orders or 10 orders. For these 10 orders how will QR be solved for? namma metro solves this problem , when multiple tickets are bought through a same online payment apps, only one qr is generated but while scanning the ticket , the panel in the station shows which ticket is being scanned
what to do when no one is available and instructed to keep the package at the door or pickup from the door? → delivery · pickup
how to make the qr solution secure from theft and fraudulent delivery partners ? we can make one qr per waybill , like when the order is placed instead
how to make the qr without friction ? by sending the picture
is the delivery route optimized for the delivery agent? they are dynamically route optimised
verify if for COD payment otp is needed or not? yes if the aov is high and if the item has more value than the prize tag on it · or open box delivery
what if the camera of the delivery guy is not working? can not happen because he has to scan the barcode of the parcel regardless
what if the screen of the reciver is cracked or shattered ? pure and black QR increase to increase contrast · padding between the UI and QR for better distinguishing · Lets say the users phone has that green line on his phone, still QR works since it will have maximum contrast · even if the screen is crack then make modules of the QR bigger · we will use Level H QR. to maximize error correction
why is a guy doing a cod and asking to keep it at the doorstep? assumption - cod is usually done when the user does not trust the website , when the delivery is dispatched from the website , the user builds trust on the website

details embedded in the qr → "QR - Qr will work even when the screen is shattered or during gl[are / bad] weather condition - So we will use pure and black QR · increase the padding between the UI and QR for better distinguis[hing] · Lets say if the users phone has that green line on his phone, still [works] since it will have maximum contrast · even if the screen is crack then make modules of the QR bigger · we will use Level H QR. to maximize error correction"

2.3 "madam otp" map (extended version)

Same branches as Map D in 03 Whimsicals Mind Maps, with these additions on the "exact ETA" and "multiple orders" nodes:

2.4 "why returns" and "how to predict the right ETA?"

2.5 "4th rail" map

2.6 Normal flow of delivery (day by day)

on 1st , user placed the order (delivery is scheduled on 4th day ) (protien powder from flipkart ) 2nd day shipped to the whearhouse locally 3rd is dispatch 4rth morning is out for delivery and the driver is assigned by end of the day driver will deliver the order , driver will stack the order based on the optimised route by the application driver will call the user and check 1) buyer is available and picks up the call -simple solution (otp and done) 2)buyer is not availalable but picks up the call - a) reschedule it to some other day b) some one else is picking up in his place (some one else is there and there is friction as he is not there and he picks up the call ) 3)buyer is not available and does not pick up the call - (NDR has to be filed as the customer is unavailable )

Flow map: day1 user placed the order (protien powder · flipkart · 4 days delivery) → day2 and 3 shipped and dispatched → day 4 - morning out for delivery and the driver is assigned → driver will place the order based on the optimised route in a stack → - 2) buyer is not availalable but picks up the call → a) reschedule it to some other day · b) some one else is picking up in his place → OTP is shared / OTP is not shared (buyer is not picking up the call) but otp is shared later · c) leave it at the door/ security - shares OTP - 3) buyer is not available and does not pick up the call - cod and prepaid - (NDR has to be filed as the delivery is failed ) → recheduled to the next day - 1) buyer is available and picks up the call - simple solution (otp and done)-otp is shared - 4) cod (high AOV)- OTP +money - delivery guy doesn't know the location → ask on whatsapp/ call · mark as customer unavailable/ other issue

2.7 Return (day by day)

2.8 Voice agent and logistic agent contribution (purple and green cards, verbatim)

voice agent contribution: calling before hand for availability of the user sentimental analysis of the user (if he seems confused or not) having the data about his specific order understanding the state of the user. 1) if he/ alternative people are available, and leave at the door etc trying to decode what the user is saying in low connectivity and noise environment

(arrows:) context about the order is sent to the voice agent through logistic agent · optimizing the delivery guys's route based on the users current status if he is busy etc

logistic agents contribution - generating dynamic QR per order ( fraud and friction ) QR solves for fraud and friction Sending a trigger to the voice agent to make a call to the user qr solution- this has to be a picture sent to the customer or his proxy in the messages or whatsapp and not a link that needs to be clicked optimized dynamic route updating the Live location from Wa to delhivery's users history to permanently save the users location 1. leave at the door only - the picture of the parcel at the door (if the users tells to leave the parcel at the door when are we going to ask for alliterative numbers - whatsapp based question, or if there are multiple orders for the same customer , the bills will be combined into a same qr and is added into the metadata , the delivery guy will scan each barcode on the box and scan the qr simultaneously for every package

logistic agents for returns lets take a picture of the surrounding

Label on board: "Acts on its own inside limits somebody set. The person deals with whatever falls outside those limits." (the L3 definition)

Voice agent call flow (small mind map): call placed through voice agent - voice agent will call 2 times → - pick up → get the details from the user with 100 percent confidence → does the task as instructed · less than 80 percent confidence of the ai agent → delivery guy will call again to confirm - unhappy flow → 1st call → if doesn't pick up → call again - 2nd call → if doesn't pick up → delivery guy will call again manually · if he does not pick up

2.9 Agent - human interaction flow (framework, verbatim)

framework to think about agent human interaction

raj has ordered a pair of shoes which he expecting on day 3 and arjun is the delivery agent who will delivery him the show. this flow will help you understand how does the agent works

  1. How the user initiates interaction.-first raj gets a whatsapp message sent by the logistic agent that if he is available or if there are any alternative people who will be available to pick up the order. This is where Raj then gives his mom's contact.

  2. How the agent communicates with the user. - through voice agent - a call is placed (this called is shown in the called id as "delhivery agentt")

  3. What information the delivery agent sees. - if raj is available or not with a confidence score. if the if the confidence score is less than 80%, then the delivery agent will have to call again and confirm.

  4. When does the agent asks for approval. - a)To send the phone number of alternative people who will be available to collect the order or delivery Ask him to approve.

b)Approvals for Arjuns Gps (geofence)

c) to get the current data of the order that arjun has placed for a personalized voice call

  1. When the agent acts independently.: when the confidence score is high from the voice agent that Raj or his mom is a) available to collect the order (after almost 2 calls are placed) b) not available to collect the order (after atmost 2 calls are placed

  2. How the user handles exceptions or escalations. If the confidence score is <80% If, due to background noise or low connectivity, the voice agent is not able to determine the exact status, then the delivery agent Arjun does a manual call.

  3. how does the agent interact with delivery agent If, let us say, The voice agent gives a low confidence score that Raj is available or not available. The dynamic route is not going to get changed, and Arjun, regardless, will still come to Raj and then place a manual voice call to confirm if he is available or not available, or leave at the door.

If there's a high confidence score (95% to 100%) that Raj is not available, then the logistic agent will dynamically reroute Arjun's route, will mark him as not available, and will move to the next delivery address after raj

Touchpoints (small map): delivery agent - H1 → point of touch - driver app/courier app → automatically and dynamically reroute his delivery route […] · end customer - (buyer/ alternative reciever) - H2 → point of touch - call/ whatsapp (voice rail) → first interaction is for […] triggered in the morning "[…] out for delivery" · ask: if you are not ava[ilable today, can you give us] alternative numbers w[hom we can contact for delivery] · voice agents calls the […] buyer first → if he does not pick up → alternative number provided in the first interaction → send the qr to either of these alternative contacts: mom · security · maid · neighbor · "send him a messages on his delivery app to call for low confidence of the voice agent (if the voice agent ain't that confident of status of the customer)" · "QR will sent as an ima[ge on whatsapp to the buyer/] alternative reciver" · "voice agent script (something like this): hey vardhan , your order for shoes is scheduled today and the agent will come to your door in an hour , are you available or any one else is available on behalf of you , should we leave it at the door?, reschedule it some other time you are available today or tomorrow ?" · "why voice agent will have a confidence level that then scores how confident it is that the end user is actually not avialable because we want to reduce postive and false negative[s]"

2.10 Stage flow charts (the white flowcharts on the right of the board)

flowchart TD
    subgraph S0[Stage 0 - Payment Setup]
        A0[Buyer selects Agentic COD at Flipkart checkout] --> A1[UPI Reserve Pay - SBMD - hold placed]
        A1 --> A2[Hold token bound to shipment AWB]
    end
    subgraph S1[Stage 1 - Alternate Capture]
        B0[Order marked out for delivery - morning WhatsApp] --> B1[WhatsApp: any alternate numbers?]
        B1 --> B2[Buyer registers alternates - mom, security, maid, neighbour]
    end
    subgraph S2[Stage 2 - Contact and QR Issuance]
        C0[Stack reaches T-5, five stops out] --> C1[Voice agent Call 1 to buyer]
        C1 --> C2{Buyer answers and confirms?}
        C2 -- Yes --> Q1[Logistic agent generates QR tagged to buyer]
        C2 -- No pickup, or unclear audio --> C3[Hold - stack keeps advancing]
        C3 --> C4[Stack reaches T-3, three stops out]
        C4 --> C5[Voice agent Call 2 - status check]
        C5 --> C6{Confirmed now?}
        C6 -- Yes --> Q1
        C6 -- No --> C7{Confidence buyer unavailable >= ~70-80%?}
        C7 -- Yes --> C8[Driver app nudges DA to call - route not paused]
        C8 --> C9{DA confirms recipient present?}
        C9 -- Yes --> Q1
        C7 -- No --> D1
        C9 -- No --> D1[Voice agent calls alternates from Stage 1 list]
        D1 --> D2{Alternate answers and accepts?}
        D2 -- Yes --> Q2[QR generated, tagged to that alternate]
        D2 -- No --> E[Escalate - see unhappy flow]
        Q1 --> QS[QR sent as WhatsApp photo]
        Q2 --> QS
    end
    subgraph S3[Stage 3 - Doorstep Handoff and Settlement]
        F0[DA enters 15m geofence - Agentic Safe Drop unlocks] --> F1[Scan order barcode]
        F1 --> F2[Scan recipient QR]
        F2 --> F3{Metadata matches authorised recipient?}
        F3 -- No --> F4[Reject and flag as fraud]
        F3 -- Yes --> F5[Camera PoD photo + GPS proximity confirmed]
        F5 --> F6[Pine Labs capture fires on UPI Reserve hold]
        F6 --> F7[Payment cleared - DA app flashes Safe to Leave]
        F7 --> F8[Order logged as delivered - buyer gets receipt push]
    end
    S0 --> S1 --> S2 --> S3

2.11 Field photos on the board

The board also holds screenshots and photos of a real delivery agent's phone: a Delhivery tracking page in WhatsApp's browser ("Arriving Today", Delhivery Maps), a delivery-rider rating screen ("Your Live Rating"), and the rider console showing "Contact consignee is pending" and "Geofence not [reached]" on a shipment card. Your label: "Mind map". These are the field evidence for the Evidence criterion; keep the originals in the folder.

Back to 00 README Index