You're right that a lot hinges on the details, but I don't see how collecting personal information (like customer name, etc) from a customer isn't considered collecting consumer data, unless it's specifically carved out in the rules.
That said, your medical example may actual be a better one than my original. AI systems are already better than human doctors at quick diagnosis from medical reports - some studies showing 2 or 3x better than a human doctor - and use of AI as a part of triage in the emergency room is one of the most obvious positive applications of AI.
An opt-out system that is 2 or 3x worse at diagnosis in medical emergencies - I think we both agree that's the definition of what this bill is supposed to help prevent. But that won't be possible very soon. Having an AI system that is more accurate at diagnosis than a human doctor creates a potential liability for the hospital, since they can't provide equal service to someone who opts out of interacting with the AI - likely in terms of both speed or accuracy.
I don't see a solution to that other than to limit the capability of the AI system you have in the emergency room, so that the human capability caps what the AI is allowed to do? Is there another approach? The only other one I can come up with would be to create an exemption in the rules that allow there to be differences in the two systems when there is no human equivalent... but that will very quickly be every AI system, and so the rules essentially stop working.
Regarding the 4 mph car - it can be useful, sure. But it's slower than every other form of transportation they had at the time, and the three-person requirement to drive any car made it much less practical than existing forms of transportation. Horses in carts are faster, require less people, and have a lower operating cost - but only because of the restrictions. Trains are faster, carry more load, and can go longer distances. The laws made it very unlikely that the average worker would be able to buy a car and hire a three man crew, so they could drive slower than a horse.
When I was researching it, the general historical consensus was that the law's intention was to make automobile ownership impractical, not enforce a practical need. You could race them on tracks. That was really their main use during that time, and certainly the mass market functions were limited by the laws in ways it wouldn't have been without them. Car ownership in the 1850s was not an everyman activity, for a number of reasons.
