Some people do care about your ex work places or your degree and alma mater. But it's true -- that's not what really matters.
You'll have a much more interesting career (and life) if you try different things and follow your gut. You don't need to check all the boxes that everyone else does to succeed.