[Fixed!] Value betting slow / not working?

Ok, so… First of, @Hanna, thanks for reaching out via email, I appreciate your handling of the issue considering the post was made late Friday evening. While I generally agree with KIMMYK’s post that you replied to, I do realize that you’re a small company and won’t necessarily have support staff on deck for individual service 24/7.
However, I don’t get how no one was around to fix the issue all weekend, or at least communicate that you were aware of it. Do none of you guys use the service yourself? That seems unlikely if it’s anywhere near as good as described, so why didn’t anyone spot this on Friday already…
@Simon, thanks for the fix, it worked after some initial wonkiness (I think it loaded some old odds but it corrected itself after a bit). But as you can see from the recent replies, it’s down again.

So Friday and Sunday down but Saturday (peak load, I assume) no problem. No communication on this post so I assume it sorted itself out somehow…? Fixed monday morning, then crash again tuesday afternoon. Fixed pretty quick (still at the office?) but then down again already on Wednesday afternoon. While I took a deep breath before writing this reply it started working again - by itself apparently since there’s been no comment from employees here telling us it’s fixed.

I’m having a hard time coming up with a reasonable explanation for that timeline. The service is a few years old, so the frontend should be working by now, at least better than this… Peak load doesn’t seem to explain it since it works on a Saturday but crashes on a Tuesday and Wednesday. Neither does problem with hosting since 4 days with downtime in the span of week would kill the reputation of any respectable host - and at the very least they would communicate something their clients could pass on anyway. The database is accessible just veeery slowly so it’s not a cable that keeps falling out.

Did someone change a setting and now everything is acting weird; has there been any update to the software in the last couple of weeks? If so, why hasn’t it been rolled back to last working backup? It’s very confusing and makes it seem kinda sketchy.

Anyway, thanks for coming to my TED talk. Hope we get a permanent fix so everyone can get on with it

1 Like

badly lagging currently again…

Yes, the service is totaly useless again, hopefully it can be fixed before Monday

Sorry guys. I agree, this is not acceptable. We will try to fix this once and for all.

I fully understand your concerns and fixing this is our top priority right now.

A workaround has been deployed now that will restart the service whenever the CPU spikes for a few minutes.

We will also add subscription days to all affected users.

The reason is not related to our servers or the load, it’s an obscure high-cpu bug that only manifests in production and never on our development servers. These kinds of bugs are sometimes hard to pin down, but we’re getting closer.

It’s also hard to send out automatic alerts. The system doesn’t go down, it just runs very slow. Will try to figure something out.

The entire team uses the service, but we’re running against the dev server with the latest beta code. Will switch to production server so we can react quicker.

I will post updates here.

Found the problem, and deployed a new version. Hopefully the slowdowns will be gone now. If not, let me know.

Technical explanation: Analyzing memory dumps I discovered a debug tool that was accidentally deployed to production, causing heavy slowdowns in certain situations.

In layman’s terms:

1 Like

Hey; It’s not working again…this morning can’t even open the page;

Update: Working now!

[error message]

2 Likes

Same here…

Sorry guys, my mistake. Don’t deploy stuff late at night without careful testing :grimacing: Should work now.

Today we will add days to your subscriptions. Thanks for having patience.