myTwin

01

Why I built it

I had a Garmin watch that gave me plenty of health and activity data, but I didn’t always know what to do with it. Seeing the numbers was one thing; knowing how to use them to plan my day was another.

At the same time I kept asking ChatGPT “What should I do today?” but those conversations didn’t have my health records, calendar or goals. I was asking for personal guidance without giving the assistant the context it needed.

That led to myTwin: a personal assistant I could talk to, backed by the information I choose to share, that turns health data, commitments and goals into a practical plan and then has a real back-and-forth about it. I built it as a solo project.

02

What it does

Meet Dash, an animated 3D companion who can look energetic, relaxed, tired or exhausted based on your estimated energy. Check in with how you actually feel and the plan adjusts.

• See your day: sleep, activity and other Apple Health records alongside your calendar.
• Understand your energy: Pro shows an estimated energy curve, possible focus times and a recap of your week.
• Plan your goals: write or speak goals for today and tomorrow; Pro finds time around existing events. Edit times, tick goals off and carry unfinished work forward.
• Rescue a busy day: preview a change to a flexible activity, confirm it, and undo it if needed. Fixed appointments stay put.
• Talk to Dash: short mentor conversations with Gemini Live voice. Speak or type; Dash can use your calendar, goals and available health data, and any calendar change suggested in chat needs your confirmation.
• Stay on track: a Home Screen widget with your energy estimate, Live Activities on the Lock Screen and Dynamic Island, and notifications for goals, events, check-ins and winding down.

A guest demo lets anyone explore Free and Pro with a fictional person, without connecting a watch or sharing personal health records.

03

How it was built

The app is Swift and SwiftUI. Apple HealthKit supplies health records and EventKit connects the calendar. Dash was modelled, rigged and animated in Blender and runs inside the app with RealityKit, with four energy states, nine animation clips and moving facial features.

For the energy model I downloaded the public PMData research dataset and trained and evaluated against 1,747 labelled days from 16 people, testing each person with a model trained on the other participants. The training and evaluation code is in the repo; the dataset is not bundled.

Regular chat can use Apple’s on-device Foundation Models on supported devices. Pro conversations use Gemini Live for replies and spoken audio, with on-device speech recognition turning the user’s speech into text. Gemini requires permission and internet access. RevenueCat manages Pro access, offerings, purchases and restores.

Honest results: the model helps track some day-to-day changes but does not improve the accuracy of exact energy scores, so the app labels its percentage and hourly curve as estimates, not medical advice.

04

Challenges

• Making voice feel natural: the microphone must wait until Dash finishes speaking so he doesn’t hear himself as the user’s answer, and the conversation has to handle pauses, typed replies and ending cleanly.
• Incomplete health data: someone may have steps but no sleep, or not enough history for a forecast. myTwin shows what’s missing and still allows check-ins and goal tracking.
• Clear calendar controls: Rescue previews before saving and chat asks for confirmation, so users always understand what will happen to their calendar.
• Demo and real Pro must match: a feature working in the demo doesn’t prove it works for a personal subscriber, so both are tested separately.

05

What I learned

• More complex models are not always better. Testing several approaches led to a small model and a clearer explanation of what its predictions can and cannot mean.
• Voice interfaces need careful turn-taking, not just speech in and speech out.
• Demo access and real subscription access need separate tests.

06

Where it stands

The project combines a custom animated character, health data, calendar planning, goal tracking, Pro subscriptions and voice or text conversations in one iOS app. Verification includes 90 passing unit tests, app walkthroughs, a personal Pro test using a simulated purchase, and Gemini conversations using fictional data.

Real-phone microphone behavior, Bluetooth and live Health syncing still need further device checks. The goal is to finish real-device testing, publish on the App Store, and keep improving it from feedback.