Most design advice assumes a precedent exists. Look at how others solve it. Follow the platform conventions. Do not reinvent the wheel. This is good advice, and it works for most interface problems, because most interface problems have been solved somewhere already and your job is to recognise which solved problem you are looking at.
Then occasionally you get handed something where that fails completely. Not hard, genuinely without precedent. I have had three of these: editor tooling inside a game engine, an automotive human-machine interface, and interaction design in virtual reality where the established patterns were mostly other people's failed experiments.
In each case I spent the first stretch doing the normal thing (surveying the landscape for prior art), and came back with almost nothing usable. Not because I looked badly. Because it was not there.
Here is what I do instead. It is not a framework. It is closer to a set of moves.
Find the adjacent domain, not the adjacent product
The instinct is to look for products that resemble yours. That is usually the wrong axis. When there is no precedent for your product, look for a precedent for the human behaviour instead.
Designing toolbar customization for the Unity Editor, the useful reference was not other game engines. It was any environment where people arrange their own workspace and then depend on that arrangement, a physical desk, a cockpit, a kitchen line. That reframe produced the finding that mattered: people who arrange a workspace are protecting a working setup, and their dominant emotion when you offer to let them change it is not excitement, it is risk. Which meant the design problem was never about the arranging. It was about making it safe to start and obvious how to stop.
No competitor teardown would have surfaced that, because the competitors did not have the feature.
When there is no precedent for the product, there is almost always a precedent for the behaviour.
Name the physics before you name the interactions
In a new domain the interactions are downstream of a set of constraints nobody has written down yet. Write them down first.
In VR the physics are brutal and non-negotiable: precision is expensive, fatigue is real, and there is no cursor. Once I had that written down for the DIRTT work, a whole category of proposals disappeared on its own. Designing in a headset, placing walls, specifying hardware, doing accurate work, was never going to survive those constraints, and no amount of interaction cleverness was going to change it. What VR is genuinely better at is understanding a space and agreeing about it with someone else. So VR became a review surface and authoring stayed where precision lives. That decision looks obvious written down. It was not obvious in the room, because the exciting version was the other one.
Automotive HMI has its own physics: attention is borrowed, not given, and the cost of a mistake is not a support ticket. Editor tooling has physics too. The user is in the middle of doing something else, always, and your feature is an interruption in someone's actual work.
The physics are where the real constraints live. Interactions that violate them fail no matter how well designed.
Build the smallest real thing, immediately
With no precedent, opinion is worthless, including yours, including the loudest person in the room. The only thing that settles an argument is watching somebody use it.
So I get to something operable absurdly early, at whatever fidelity buys the fastest answer. For the toolbar work that was an interactive prototype that simulated the Editor closely enough that people behaved as if it were real, which is the bar that matters: not whether the prototype looks finished, but whether the participant forgets to be polite about it.
And I build in the domain itself when I can. I prototype my own digital twins (a smart-home model wired to live IoT state, a persistent social environment published into VRChat), not as portfolio pieces but because being a user of the thing you are designing tooling for is the cheapest research there is. You cannot fake having been annoyed by something.
Invent the vocabulary on purpose
New domains need new words, and if you do not choose them deliberately, they get chosen accidentally by whoever wrote the first internal document, and then you are stuck with them forever.
We had a label called Editor Utilities. It was accurate and it meant nothing to anyone. In testing, half the participants read it as some unrelated developer feature and skipped straight past the thing they had told us they wanted. It became Editor Controls, and the problem evaporated. That is a rename, not a redesign, and it did more for discoverability than any visual change in the project.
Treat naming as design work with the same weight as layout. In a domain with no conventions, the words are the conventions.
Write down the rule you just invented
This is the part that separates a nice feature from a durable one, and it is the part most portfolios skip.
When you invent a pattern, you have implicitly answered a question that will be asked again, usually by someone who was not there. Who is allowed to put something in front of every user? What happens to this object when it is retired? Which parts of the interface are the platform's and which belong to whoever extends it?
On the toolbar work, the feature was a customization API. The thing that will outlast it is the governance model underneath, a stated position on who gets toolbar real estate and how that is decided. Without it, the toolbar drifts back to what it was within a few releases, and everyone involved will be surprised.
So: ship the pattern, then write the rule. The pattern solves this instance. The rule is what stops you solving it again next year.
The uncomfortable part
Working without precedent means being wrong in public more often, and it means you cannot borrow anyone's confidence. There is no article to point at, no established pattern to cite when someone asks why. You get the finding from your own testing and your own reasoning, and sometimes that is all you have.
I have come to think that is the actual skill, and it is a different skill from executing well against a known pattern. Both are real. Only one of them gets easier with experience in the way people expect.