Yes. Growing up with ties to the yakuza's bound to be more trouble than it's worth -- especially if that person is the former Fourth Chairman of the largest clan.

[Which is why in canon he tries his very best to stay away from that kind of thing, even moving away and living as an orphanage director, a taxi driver, and other kind of harmless lives. It's just too bad circumstances keep dragging him into the fray]

In this world I'll be just some random guy more.