Log Analyzer

[TBK]

So i've been working for a while on creating a tool that helps analyze datalogs. It has 2 intended audiences, and plays a different role for each one:

- The average person who doesn't know too much, and so the tool helps explain what they're looking at
- The person who knows their shit. Provide a better log analysis experience overall, as well as acts as a nice repository, log comparison tool etc...

Really wish i could rely on AI for the analysis (and to determine what to analyze), but you'll just get confident nonsense. You still need a lot of structured and contextual reasoning for something like this, and in ways AI just can't help too much.

Instead there's just a fair bit of data science powering the logic. I've also written nearly 11,000 lines of rules, weights, and inference logic. It's a framework built on multi-variate signal correlation. And that's where it gets REALLY challenging. All the context that needs to be added. And i've had help from a few people who are lot more knowledgeable than i am to refine the top end of those rules, and where the intersections get a bit tricky.

It's currently fully operational....you can upload a CSV log and it'll spit out a full analysis. But it still need work. The logic (the rules framework, which is almost 40 different tabs at this point) still needs to be refined. A small gap in 1 place creates a massive domino effect of wrong analysis.

I'm going to be occasionally sharing some (hopefully) cool things from the project here. Until Bryan comes in here and reminds me that i know nothing Jon Snow.

This is one of the cooler charts, if only because most people just don't look at this. Most logs are just WOT pulls. And most analysis is done with WOT data. This is ΔTorque/ΔPedal from an older (really terrible) tune i had.

Don't get to see how broken that is when you're looking at WOT data

Sponsored

 

[B54]

I love these types of projects. Good job! Are you planning on releasing it? And in what form? Standalone executable/script, or maybe a webinterface?

 

[Thraxbert]

Very cool project. Counting down the days until we get our first panicked post about an anomalous dot.

 

[BMWAF]

This looks really good. Good luck with development.

 

[razorlab]

Great minds think alike, I am working on something similar but a bit different.

https://www.supramkv.com/threads/self-tuning.15851/page-9#post-555390

https://www.supramkv.com/threads/looking-for-track-lap-datalogs.36210/

 

[TBK]

I love these types of projects. Good job! Are you planning on releasing it? And in what form? Standalone executable/script, or maybe a webinterface?

Yeah i very likely will. It started off as something i wanted for myself. I didn't want to put something out that did not make accurate/logical conclusions with context, and had enough reasoning built into it to hold back when confidence levels did no meet a certain threshold. Which necessitated a confidence threshold framework...lol you see where this is going. I think i've gone down the rabbit hole far enough at this point that i think i'm probably going to release something in the not too distant future. Web and probably app

Great minds think alike, I am working on something similar but a bit different.

https://www.supramkv.com/threads/self-tuning.15851/page-9#post-555390

https://www.supramkv.com/threads/looking-for-track-lap-datalogs.36210/

Bryan that's amazing, and in many ways, VERY similar to what i'm working on.

I can see some subtle differences. You seem to be a bit more data viz focused, and i think i'm more data viz plus analysis focused. The analysis part is essentially where i've been spending 90% of the time, because it's also largely platform dependent obviously. So each platform (e.g. B58, EJ257 etc...) is basically like an entirely separate and standalone product that requires it's own massive reasoning framework. Different rules, thresholds, failure points etc...Those frameworks, considering the time investment required to build them, would create a very safe moat over time.

The reason i see the analysis part as essential is because i think it's not just time consuming, but something just way too hard to spot certain essential things in the data ourselves. Some of that is sorted by making sure the right charts/graphs are in place, but that doesn't help debug some tune issues for example.

You're also lap focused. (what you're doing there looks amazing btw)...and i actually do have something like that in the pipeline. But i want to invest more time into the analysis framework before going that direction. Depth over breadth at this point, basically. The amount of up-skilling i've had to do....oof. But i'm nothing if not obsessive.

just as an example, and i know i'm not saying anything you don't know, just explaining my thought process. Took the below screenshot from one of your pics. I've built so many contextualization rules before drawing any such conclusions:

E.g. was the temp recorded a spike or sustained? What was the Ambient temp + IAT delta? No ambient temp shared since it's not logged...then assume 10 seconds pre WOT temp to represent ambient here (if certain other conditions are met). Vehicle speed / airflow to understand if we're looking a heat soak vs actual cooling system limitations. I'm oversimplifying it basically, but i'm trying to explain where a lot of my time has been invested.


I've also been building a mod recommendation engine to alongside the log analysis. Of course that would be predicated on the log user including what basic mods they've already done (e.g. intake manifold, catless DP etc...). Another massive framework of rules. Didn't mention this earlier, because this isn't the meat and potatoes, but i def see this as a strong magnet. Thousands of lines of types of rules across a number of tabs i've lost count of by now. By platform of course. The layering of triggers and thresholds is what protects the recommendation engine from making silly recommendations

I would've said let's just combine forces and maybe even take this seriously (esp since if my reading is correct, and going by the color palette you're using alone, you have a creative background, and i have a generalist + specialist performance marketing/analytics one) but you might be even more misanthropic that me haha. Well, at least online.

 

[razorlab]

Yeah i very likely will. It started off as something i wanted for myself. I didn't want to put something out that did not make accurate/logical conclusions with context, and had enough reasoning built into it to hold back when confidence levels did no meet a certain threshold. Which necessitated a confidence threshold framework...lol you see where this is going. I think i've gone down the rabbit hole far enough at this point that i think i'm probably going to release something in the not too distant future. Web and probably app





Bryan that's amazing, and in many ways, VERY similar to what i'm working on.

I can see some subtle differences. You seem to be a bit more data viz focused, and i think i'm more data viz plus analysis focused. The analysis part is essentially where i've been spending 90% of the time, because it's also largely platform dependent obviously. So each platform (e.g. B58, EJ257 etc...) is basically like an entirely separate and standalone product that requires it's own massive reasoning framework. Different rules, thresholds, failure points etc...Those frameworks, considering the time investment required to build them, would create a very safe moat over time.

The reason i see the analysis part as essential is because i think it's not just time consuming, but something just way too hard to spot certain essential things in the data ourselves. Some of that is sorted by making sure the right charts/graphs are in place, but that doesn't help debug some tune issues for example.

You're also lap focused. (what you're doing there looks amazing btw)...and i actually do have something like that in the pipeline. But i want to invest more time into the analysis framework before going that direction. Depth over breadth at this point, basically. The amount of up-skilling i've had to do....oof. But i'm nothing if not obsessive.

just as an example, and i know i'm not saying anything you don't know, just explaining my thought process. Took the below screenshot from one of your pics. I've built so many contextualization rules before drawing any such conclusions:

E.g. was the temp recorded a spike or sustained? What was the Ambient temp + IAT delta? No ambient temp shared since it's not logged...then assume 10 seconds pre WOT temp to represent ambient here (if certain other conditions are met). Vehicle speed / airflow to understand if we're looking a heat soak vs actual cooling system limitations. I'm oversimplifying it basically, but i'm trying to explain where a lot of my time has been invested.


I've also been building a mod recommendation engine to alongside the log analysis. Of course that would be predicated on the log user including what basic mods they've already done (e.g. intake manifold, catless DP etc...). Another massive framework of rules. Didn't mention this earlier, because this isn't the meat and potatoes, but i def see this as a strong magnet. Thousands of lines of types of rules across a number of tabs i've lost count of by now. By platform of course. The layering of triggers and thresholds is what protects the recommendation engine from making silly recommendations

I would've said let's just combine forces and maybe even take this seriously (esp since if my reading is correct, and going by the color palette you're using alone, you have a creative background, and i have a generalist + specialist performance marketing/analytics one) but you might be even more misanthropic that me haha. Well, at least online.

Does your analyzation of EJ257 always end in a blown up engine because that is always the end?

Oh, I am 100% doing analyzation + coaching/suggestions. That just isn't forward facing yet.

There are some hints with bones of it in the app already though:

And of course, the ever elusive gate-kept "what tables do I need to change/tweak to get rid of this torque/fill limit":

 

[TBK]

Does your analyzation of EJ257 always end in a blown up engine because that is always the end?

Oh, I am 100% doing analyzation + coaching/suggestions. That just isn't forward facing yet.

There are some hints with bones of it in the app already though:

And of course, the ever elusive gate-kept "what tables do I need to change/tweak to get rid of this torque/fill limit":

That looks really good. Are you keeping it B58 focused? Each new platform is a ton of work, and a part of me is wondering if starting with more than 1 platform is even the right play. For now me thinks it makes sense. The B58 may be popular, but i don't think it's enough to help get something like this enough traction to get off the ground.

I still have a piston 4 with a cracked ringland as a living room ornament. All the way from 2011. Maybe one day when i have enough $$ i'll go back and pull of a Dewy type build, which is just as nice as it gets imo.

 

[razorlab]

That looks really good. Are you keeping it B58 focused? Each new platform is a ton of work, and a part of me is wondering if starting with more than 1 platform is even the right play. For now me thinks it makes sense. The B58 may be popular, but i don't think it's enough to help get something like this enough traction to get off the ground.

B58 and S58 as they are very very similar. The track analyzation part is platform agnostic, of course.

I still have a piston 4 with a cracked ringland as a living room ornament. All the way from 2011. Maybe one day when i have enough $$ i'll go back and pull of a Dewy type build, which is just as nice as it gets imo.

Dewy goes through a ton of engines so not sure if that is a good benchmark, but to be honest, that is just Subaru life. Glad I left that world in the past. I'm also very biased, as Mitsubishi was my life for 15+ years.

 

[TBK]

B58 and S58 as they are very very similar. The track analyzation part is platform agnostic, of course.



Dewy goes through a ton of engines so not sure if that is a good benchmark, but to be honest, that is just Subaru life. Glad I left that world in the past.

Does he? Even with that fancy IAG block? Or is the new one he's using more robust? That car just has my heart.

B58/S58 start makes sense. I'm going a bit wider that that to start. I'm also looking into multiple logging platforms. It's not a big deal, but standardizing the naming convention is def something that needs to happen in the process. I guess in your case it's just BM3....so annoying....




Signals:

• Boost (Pre-Throttle) [psig] = turbo outlet pressure (before throttle)
• (RAM) Boost Pressure (Manifold Target) [psig] = ECU dynamic target
• (RAM) Boost (MAP) [psig] = actual manifold pressure (engine sees this)

Interpretation:

• For engine load / torque delivery → compare MAP vs Manifold Target
• For turbo / boost control behavior → use Pre-Throttle Boost + WGDC

Boost control diagnosis:

• Use MAP Manifold Target to determine if requested load is achieved
• Evaluate tracking accuracy, stability, and overshoot

Turbo / hardware diagnosis:

• Use Pre-Throttle Boost with WGDC
• If WGDC is high but Pre-Throttle is low → turbo-side issue (leak, actuator, inefficiency)
• If Pre-Throttle is high but MAP is low → airflow restriction (throttle closure, intake limitation)

Cross-signal context (allowed and useful):

• Pre-Throttle vs MAP → diagnose throttle behavior or intake restriction
• Do not use cross-signal comparisons to judge boost control accuracy

Throttle insight:

• Pre-Throttle >> MAP = throttle limiting airflow, not a boost deficit

Other signals (conditional use):

• Boost Pressure (Target) may be static or non-dynamic
• Only use if validated:
  • Confirm it updates dynamically with RPM/load
  • Cross-check that it tracks closely with Manifold Target under load
• If it does not track → ignore for analysis

Priority:

• Prioritize MAP vs Manifold Target accuracy and stability over peak boost values

 

[razorlab]

Does he? Even with that fancy IAG block? Or is the new one he's using more robust? That car just has my heart.

B58/S58 start makes sense. I'm going a bit wider that that to start. I'm also looking into multiple logging platforms. It's not a big deal, but standardizing the naming convention is def something that needs to happen in the process. I guess in your case it's just BM3....so annoying....




Signals:

• Boost (Pre-Throttle) [psig] = turbo outlet pressure (before throttle)
• (RAM) Boost Pressure (Manifold Target) [psig] = ECU dynamic target
• (RAM) Boost (MAP) [psig] = actual manifold pressure (engine sees this)

Interpretation:

• For engine load / torque delivery → compare MAP vs Manifold Target
• For turbo / boost control behavior → use Pre-Throttle Boost + WGDC

Boost control diagnosis:

• Use MAP Manifold Target to determine if requested load is achieved
• Evaluate tracking accuracy, stability, and overshoot

Turbo / hardware diagnosis:

• Use Pre-Throttle Boost with WGDC
• If WGDC is high but Pre-Throttle is low → turbo-side issue (leak, actuator, inefficiency)
• If Pre-Throttle is high but MAP is low → airflow restriction (throttle closure, intake limitation)

Cross-signal context (allowed and useful):

• Pre-Throttle vs MAP → diagnose throttle behavior or intake restriction
• Do not use cross-signal comparisons to judge boost control accuracy

Throttle insight:

• Pre-Throttle >> MAP = throttle limiting airflow, not a boost deficit

Other signals (conditional use):

• Boost Pressure (Target) may be static or non-dynamic
• Only use if validated:
  • Confirm it updates dynamically with RPM/load
  • Cross-check that it tracks closely with Manifold Target under load
• If it does not track → ignore for analysis

Priority:

• Prioritize MAP vs Manifold Target accuracy and stability over peak boost values

I’m not even bothering with BM3 since it’s such a hacky thing uplifted by marketing. I personally made a decision to not spend anymore effort with it.

 

[TBK]

I’m not even bothering with BM3 since it’s such a hacky thing uplifted by marketing. I personally made a decision to not spend anymore effort with it.

Fair enough. I'm doing because BM3 users still represent a fairly large contingent.

 

[razorlab]

Fair enough. I'm doing because BM3 users still represent a fairly large contingent.

Yea I still tune plenty of people on BM3 but it’s always such a pain compared to other platforms.

I’m building my app with love labor and I do not love BM3.

 

[MisterSkiz]

You guys vibe coding this or doing some good 'ol fashioned developing?

 

[noogie]

got github link?

Sponsored