Not really. If a team has an xg of 2 but they are all from defenders then that will say that team should've scored two, but that would not be the case because those chances fell to players who aren't as good at finishing as attackers, so that's where it falls apart.
Two things there.
1) there's something wrong with a team whose best chances are consistently going to defenders
2) this shows that they're bad at finishing
It doesn't "fall apart". It's a valuable stat without being absolute gospel, because no one stat is.