Help show Consumer Reports how it's done

[TaVern]

Originally Posted by mkaresh

I run various checks on the responses that are submitted and the user-data collected in the process. Those that fail these tests are investigated. So far, very few such checks have been required, and so far all errant entries discovered have been caused by human error, not malicious intent.

I appreciate your reply.

How is it determined what is human error and what is malicious intent? Could you please elaborate on what kind of "checks" are performed? Thanks in advance.

[concorso]

Originally Posted by StealthV

Really want to poke the eye of GM with a sharp stick? Rate the quality of the dealerships as well as the effectivity of marketing campaigns compared to the competition.

There's a reason people buy Yota$ and it's not because they build an exciting, quality car. Yota$ are rubbish supported with good marketing and dealers. Bunch of lemming$ on the Yota


P.S. GM and its dealers haven't figued it out yet.

My father has owned 3 Escalades in a row, plus he's on his 2nd Corvette, all bought new...and the last time the 'Lade was in for service, he got a base Cobalt loaner. He's not one to complain, but when we left the delearship, he said "They couldn't find a base Lacrosse anywhere on the lot?". But when their Lexus ES was in for transmission slippage, they got an RX as a loaner, without asking! My father will always buy GM products, regardless of what the buying/owning experience is like, but I cant see them stealing many people with this kind of service. I really think Cadillac needs to take Buick and separate from the rest of the GM lineup. IF they truly want this brand to move upscale, they need to fool the same people who believe the YotaADtrain, into thinking that Cadillac has no relation to the rest of GM...

[mkaresh]

Originally Posted by TaVern

I appreciate your reply.

How is it determined what is human error and what is malicious intent? Could you please elaborate on what kind of "checks" are performed? Thanks in advance.

I'd rather not elaborate, for the same reason you won't get a detailed description if you ask a bank about their security procedures.

The main deterent, I suspect, is that trying to muck up the results would require too much effort, and have too little impact. It's hard enough to get people to provide data for cars they do actually own.

[mkaresh]

Originally Posted by gothicaleigh

So should I only be paying Sal for advertisement once I make what I consider a profit from his site each month? Your site is profiting from exposure here. What is more, it is absolutely dependent upon sites like this supplying the information you deal in.

Perhaps it is time you started paying for the privelege just like the rest of us.

Being a supporting member here only costs $5 a month. No more excuses please.

It's not just a matter of this forum and $5 a month. I'm not going to cut special deals with any forums. If and when I start paying one forum, I'll pay all of those that are assisting me.

I've thought for some time that this change will happen when I start charging non-participants for access to the results. The difference between me and you is that I'm not currently looking for any revenue from anyone who reads my posts here. If, like a vendor, I was looking for money from people here, then you're correct that I should be considered a vendor. But currently I am not.

This is an explanation, not an excuse. An excuse is when someone's done something wrong, and there is no valid explanation. I've done nothing wrong.

Sal is an intelligent adult capable or running his own enterprise. I asked for his help getting this research off the ground, and he agreed. As I've said many times, I'm very grateful for his help.

This wasn't contingent on results being ones that would make everyone happy. The goal was to provide results before anyone else, and credit this forum for its assistance, and I fully delivered on this.

[dqw1]

It's only 5 dollars a month not 50.

[TaVern]

Originally Posted by mkaresh

I'd rather not elaborate, for the same reason you won't get a detailed description if you ask a bank about their security procedures.

The main deterent, I suspect, is that trying to muck up the results would require too much effort, and have too little impact. It's hard enough to get people to provide data for cars they do actually own.

Once again, I appreciate your reply.

Please understand, I would never ask a bank about their security procedures if they were FDIC insured, since my deposits would be guaranteed. I could not find any guarantee on your website that the user submissions you publish are true or even accurate. Therefore, I feel obligated to address some issues.

I was initially skeptical, but kept an open mind to your website despite the comments in this thread. I enrolled myself into truedelta.com, to research further since your website requires a valid email address to gain access beyond the "sample" data that is available to those who are non-members.

I was looking to reassure myself of the high-quality "results" you proclaim before surrendering my vehicle history to you. I have since then requested my email address be removed.

I found several points of concern. Because I am a Cadillac owner, I naturally directed myself to your Cadillac reviews. Under "Vehicle Repairs History", I found the 2008 Cadillac CTS categorized as a 2WD option, an AWD option, and a "4WD" option. There was even a "Vehicle Repairs History" for a 2008 Cadillac CTS "4-cylinder".

You state these records are verified. And even if they are human-error, I have to believe your data is tainted across the board, as you list dozens more vehicle manufacturers with even more unknown errors. Furthermore, the malicious-intent aspect seems even more complicated.

Originally Posted by mkaresh

All actual samples are flawed. Even with random sampling, the response rate is never close to 100% for this sort of research.

Just because the sample isn't random doesn't mean the research is not "scientific."

Two things control for the non-random sample:
1. People only report what happens after they join.
2. I'm not asking about opinions, but about whether or not an event happened.

There was another submission of a 2005 Cadillac CTS 6-cylinder automatic 2WD that contained a 337 word 'rant' of one repair incident. This leads me to believe your website is nothing more than a "dear diary" of upset car-owners. To describe this submission alone as "scientific" is deceptive.

Your website also promises that "non-members will pay $24.95 per year for this access; you [those participating] won't". This clearly hopes to be a money-making operation for you.

Generally, these concerns would not trouble me in the least. However, you have advertised yourself on this forum as more accurate than Consumer Reports with the confidence and security measures of an FDIC insured banking institution. I find these tactics dishonest and fraudulent.

More importantly, you've done all of this without giving a supporting membership to the CadillacOwners.com Forum members who undoubtedly and unknowingly assisted you with this betrayal and unethical display of business practices.

I encourage you to direct everyone to your website, as they will see for themselves everything I have described above as being true.

[RightTurn]

:golf clap:

[Brett]

Non-probability (non-random) samples:

These samples focus on volunteers, easily available units, or those that just happen to be present when the research is done. Non-probability samples are useful for quick and cheap studies, for case studies, for qualitative research, for pilot studies, and for developing hypotheses for future research.




Your system is flawed. The reason random sampling is used in these types of surveys is that the sampling error can be mathematically calculated. Your data is useless. Your site should contain the phrase "For Entertainment Purposes Only"

Originally Posted by mkaresh

All actual samples are flawed. Even with random sampling, the response rate is never close to 100% for this sort of research.

Just because the sample isn't random doesn't mean the research is not "scientific."

Two things control for the non-random sample:
1. People only report what happens after they join.
2. I'm not asking about opinions, but about whether or not an event happened.

If this were an opinion poll, it would be much more crticial to have a random sample. Consider the difference between asking people "is it raining" vs. "are you having a good day?" Even better, having people sign up today, then report whether it rains during the following week.

My results end up very similar to those of other surveys. They just include more detail, and are updated much more frequently and promptly.

[mkaresh]

There is absolutely nothing deceptive or unethical in anything I have posted here or on my site. No betrayal, either. That's your own bias talkiing. You're seeing what you want to see, not what's actually there. I have been delivering exactly what I promised to deliver.

Amazingly, you've managed to take things I've done to be helpful and above reproach, and turned them into negatives. Just incredible.

Let's start with charging $24.95. It's in the future tense, and has been for a few years now. If you look in the OP to this thread, I mention that an advantage of participation is free access to the results. The obvious implication: others will eventually have to pay. This has never been hidden, in the slightest. It's in the OP, then it's on the page that the OP links to.

But let's consider a few things--assuming your mind is half as open as you think it is:

First, I'm not currently trying to charge anyone. I want the value to absolutely be there first, and my priority is on the research, not revenue. Just about anyone else doing this would have shifted the focus to revenue some time ago. Not many results? So what, charge people anyway, that's what most people in my position would do. But I don't. Instead, I'm giving everything I gather away for free to anyone who helps out by participating.

No one else does this. Not CR. Not JD Power. Yet I'm the bad guy, because in the future I hope to charge those who don't want to help make the results possible, but want access anyway.

Why mention the $24.95 now, if I'm not offering this option yet? Because it helps people put a dollar value on their participation. Only about one-tenth as many people signed up each day before I added that line. It's about getting results--which I give away for free--not earning money.

Second, I recently started posting every single repair response to the site. I thought I was doing a good thing, being transparent and providing people with as much detail as possible.

But here you are, turning this against me. Do you think people would be better served if I hid all of these resposes, like CR and JD Power do? I guarantee that their responses are at least as messy as mine. It's people like you that make organizations like mine conceal data that might help people. If there are enough people like you, I might have to do what others do, and stop displaying individual responses.

These responses are posted as soon as they're submitted, so the newest ones haven't been through the error-checking process yet.

People are people, and they make mistakes when filling out surveys. Any surveys. I frequently tweak the surveys to reduce the error rate, but zero errors is an impossible goal. No one achieves this, any more than anyone builds entirely problem-free cars.

The specific data you have focused on are for a separate survey on bodystyle and powertrain I generally haven't bothered to error-check, because in most cases those data aren't used in the analysis. I don't currently report results separately by engine or drivetrain for the CTS, so this information is not critical. When information is not critical, I don't email members to get it cleared up.

What is critical? The repair submissions. Every single one of those is reviewed. Where there are errors, and the correct information is not obvious, those members are emailed for a clarification.

Normally people are in favor or me keeping the number of emails to a minimum. But it seems you'd rather me email all of these people, even if the correction is not essential.

But you might be right, in a way. Perhaps I ought to clean that data, since I'm now posting that information to the site (I wasn't a couple months ago), and people will assume that if there are errors in things like cylinder count there will be errors elsewhere as well. It's not evident that critical data are error-checked thoroughly, while non-critical data are not.

I should also note that only vehicles for which repairs are reported are posted on the "repair histories" page. Otherwise the page would, in most cases, be cluttered with repair-free cars.

Every month about 500 repairs are reported by over 6,000 participants. You take a single incident, and reach a general conclusion that all 6,000 are there just to complain from that? If so, why are over 90% of people reporting no repairs each month? For some models, hardly any repairs have been reported, even with dozens of owners participating.

The actual data overwhelmingly reject your conclusion that people are just there to complain because they are upset.

In the case of the 2008 CTS, many, perhaps even most participants signed up before they even took delivery of the car, when they have extremely positive feelings about the car.

You're lecturing me on being "scientific." Do you think you're being scientific?

You're being the absolute opposite, picking isolated examples out of hundreds because they seem to provide evidence for what you want to believe.

How do you like it when a domestic car hater learns of a single problem someone has had with a domestic car, or perhaps a couple of problems a couple people have had, and then concludes from this that all domestic cars are junk? Well, that's exactly what you're doing here with my research.

And when that doesn't do the trick, you simply make stuff up. Like saying "you have advertised yourself on this forum as more accurate than Consumer Reports with the confidence and security measures of an FDIC insured banking institution."

I said that my reasons for not divulging all of my security measures were the same as those a bank would have. Interesting how you managed to twist this into my supposedly saying that I use the same security measures as a bank. Frankly, that would make no sense.

Nowhere do I say my results are more accurate than CR's. Not here, not on my site, not anywhere else. I've said their methods are flawed, and they are. But their sample sizes are in most cases much larger, and this currently compensates for their inferior methods in terms of accuracy.

If my sample sizes were as large as CR's, my results would be more accurate. But we're not there yet, and I have never claimed to be. I've claimed that my methods are superior, and they are.

In the OP I made two specific claims: that I would post the actual repair rates, not just dots, and that I'd have results months sooner.

I have delivered on both claims, in full.

I am glad to hear you removed yourself from the panel, as you signed up with an agenda, not with an honest desire to contribute. In general, I hope that those who start with conclusions rather than evidence will not join.

I believe that both manufacturers and car owners will be best served by starting with evidence, rather than conclusions. In recent years, the domestic car makers have been hurt more than helped by those who insist on picking and choosing their "evidence" based on what they want to believe. The answer is not to do the same, just in favor of the domestics.

[mkaresh]

Originally Posted by Brett

Non-probability (non-random) samples:

These samples focus on volunteers, easily available units, or those that just happen to be present when the research is done. Non-probability samples are useful for quick and cheap studies, for case studies, for qualitative research, for pilot studies, and for developing hypotheses for future research.

Your system is flawed. The reason random sampling is used in these types of surveys is that the sampling error can be mathematically calculated. Your data is useless. Your site should contain the phrase "For Entertainment Purposes Only"

I explained in the post you cited why a random sample is less essential for the research I'm conducting. It would be better to have a random sample, but it is not essential.

These aren't finer points you'll find covered in a basic stats text or class.

But out in the real world, there are no perfect samples, if only because the response rate is usually well under 100%.

Let's compare my research with J.D. Power's in a bit more detail. They're the only ones in the field that use random sampling. And this method is very expensive. One consequence: only manufacturers that can pay the big bucks see usefully detailed results. Car buyers get crumbs.

Anyway, with J.D. Power, they mail surveys to a randomly selected group of owners. These owners are asked to respond on problems that have already happened. The goal on one such survey was to have 23% of those surveyed actually respond. Are people who've had problems equally likely to respond as those that haven't? Probably not. So even while the people who receive surveys are random, those that return them are probably not. While this isn't sampling bias, it is non-response bias. There are ways to try to measure non-response bias. But good luck finding the results of any such study J.D. Power has conducted.

In my case, we do have a convenience sample. Except I don't ask people to report problems that happened before they signed up. We would have definite distortions if I did. Instead, people sign up, and then report what happens afterwards. Unlike with J.D. Power, they can't just report problems. For something to be reported, they have to have reported the problem to a dealer. And for the problem to be included in the current analysis, the dealer must agree there's a problem, and then do something that makes the problem go away. So, more than with J.D. Power (or CR), we can be quite sure there's a real problem, and not just a perceived proble.

Despite the fact that we ask separate questions and have different potential sources of bias, my results actually end up similar to those of J.D. Power. We'll probably learn in June how the 2008 CTS did on their survey. My results suggest that it'll be below average. We won't learn from them how far below average--because they don't report that information to the public.

The big differences: I provide all of the results to the public, rather than only to manufacturers who can pay millions. And I cover cars continously, not just at 90 days and the third year. And I update four times a year, not just once a year.

Ideally I'd like to have a random sample. But it's not financially feasible to have one. Even J.D. Power must severely restrict their coverage to be able to afford a random sample, even with millions in revenues from the research.

I believe that it's much better to have reasonably accurate results four times a year, on every car I can get enough responses for, for free to everyone who participates, than to make do with what JD Power and CR have been providing.

[JimmyH]

Why does anyone care about any of this? My boss has a Lexus, one of the highest rated cars out there. He takes it to the dealer more often than I take my car to the car wash.

My Grandpa used to have a Buick Roadmaster. One of CR's most problematic cars of all times IIRC. He never had a single problem with it. Not one. Not even a small one.

It's all a crapshoot folks. Buy the car you like and worry about the freaking problems later.


BTW, there is no better way to ensure people will NOT become supporting members than to suggest it to them.

[mkaresh]

Originally Posted by JimmyH

Why does anyone care about any of this? My boss has a Lexus, one of the highest rated cars out there. He takes it to the dealer more often than I take my car to the car wash.

My Grandpa used to have a Buick Roadmaster. One of CR's most problematic cars of all times IIRC. He never had a single problem with it. Not one. Not even a small one.

It's all a crapshoot folks. Buy the car you like and worry about the freaking problems later.


BTW, there is no better way to ensure people will NOT become supporting members than to suggest it to them.

The real issue isn't whether or not I'm a supporting member. It's that many people don't like the result for the 2008 CTS, and making a big fuss about whether or not I'm a supporting member is a possible way to undermine the research. If I was a supporting member, the issue would be something else, at least for most of the people raising the issue.

There's a lot of truth to the rest of your post. The main reason I started conducting this research was to show how small the differences in reliability are in most cases. CR and JD Power don't publicly release the problem rates for individual models, so small differences can seem much larger than they actually are. My solution: post the actual repair rates, not just dots.

I've pointed out many times that we're usually not talking about the number of problems per car in a given year, but the odds of having even a single problem. With just about any car there's a good chance of getting one with no problems.

This said, it's not entirely a crapshoot. You are always rolling the dice, but they aren't loaded the same for all models. Your odds of having problems are higher, in a few cases much higher, with some models than with others.

At what point should this make a difference? That depends on the actual size of the difference, and how high a priority the car buyer puts on avoiding one or two additional repairs. But each person can only sort this out for themselves if they know the actual repair rates. TrueDelta is the only one providing these.

I think this message has been lost, in part because the repair rate came in so high for the 2008 CTS that it is one of the few cases where the difference compared to most other models is substantial. But also because with CR and JD Power, all you know is how each model compares to the average, so people have come to think that "worse than average" is the same as "do not buy." When you have the actual repair rates, this isn't the case. It's like having an idiot light, vs. an instrument.

[Brett]

Originally Posted by mkaresh

But out in the real world, there are no perfect samples, if only because the response rate is usually well under 100%.

Of course there are no perfect samples, thats why Random samples are used, so that a margin of error can be mathematically calculated. There is no way to calculate your margin of error. You claim to be a PHD on your site, which means you know all this already, so that begs the question of what your actual motive is. Frankly, I dont care, I dont own a CTS.



Turn your site into a "vehicle issue" blog and monetize it that way. That could be a useful resource.

[TaVern]

Originally Posted by Brett

Turn your site into a "vehicle issue" blog and monetize it that way. That could be a useful resource.

That's exactly what this website is.

[Brett]

Well, exactly.....I should have said "market" instead of "turn into"