I have had the same issue's with either my usb or firewire connections. In my main pc I have tried several firewire pci cards of wich the ones with VIA chipsets where the worst sounding, best was (is) the one with Texas Instruments chipset but these seem hard to find these days. For years the TI chipsets where about the standard in firewire interfaces but nowadays many cards are very cheaply produced and they almost all use VIA chipsets. Another good one was the one with Ageere chipset, about as good as the TI. Any good local dealer can tell you what chipset the cards are using, if they can't just go to another store with more qualified salesmen.
With USB I have only experienced a huge diference between my laptop usb and my desktop usb. The desktop sounds much better, in fact the laptop sound quite horrible via usb despite being it a far more modern computer than the desktop is. I really don't know why this is. Maybe it a more stable power supply, maybe only the usb chipset, maybe because my desktop desn't do any speedstepping, has no dualcore etc etc, could be anything. Anyways, I'm looking for a replacement for my desktop since it's a bit at the end of it's life but I have no lue what to buy anymore, sound quality wise that is