Jump to content
LOTROCommunity
LordVorontur

Data Centre Move on 11th January 2016.

Recommended Posts

10 hours ago, nosam9 said:

Something went wrong. Apparently the game is barely playable.  They wont get it working until tomorrow.

FREELORN posted:

"We believe we have found a fix for the current login and latency issues that players are reporting with the game. However a hotfix to the game will be required. In order for us to do this we will have to build an update, deploy it to our test server, and test it extensively before we can release the fix to the live servers. This means that while we will continue to try and mitigate any issues that occur tonight, we will not be able to deploy a fix for this issue until sometime tomorrow.

We do understand this is not an optimal solution, however right now this is the only option we have. This issue was something that for whatever reason, did not appear on any of our tests including the live testing we did on Bullroarer. We apologize for any inconvenience this may cause players."

The live testing was with only one world open on the new setup, correct?  Just Bullroarer?  Not surprising that having all ten remaining worlds (plus the leftovers) active at the same time would reveal other issues.  Funny how nobody there seems to have thought of that before this.

 

7 hours ago, Dalthalion said:

At least I got what I wanted from Turbine over the winter festival.  Thousands upon thousands of free marks, scores of medallions and a goodly number of tier 1 stat tomes, all for doing a quick, emote-based "quest" for an NPC in Winter-home with all my characters.  With all the festival tokens I racked up in the process, if I wanted to get a +68% speed mount for a character, I could simply barter it.  And, for those characters of mine that haven't received the theatre titles yet, a mere 60 festival tokens got me 40TP per character, which puts me in the 29-30K area now.  And all of this was done without entering the Store or buying TP.

 

The truly pathetic thing about the festival was that no one could be bothered to put this year's variation on lotro-wiki.com.  That's pretty telling.

 

So, let them fumble around, while I play other MMOs.  They've more-or-less served their purpose this winter.

For me the focus was on essence removal scrolls - I came out with enough to see me through the rest of my playing time, or the end of the game, whichever comes first.  Like you I also got a lot of tomes, marks, and meds from the daily present sacks for a trivial amount of time.  Not that I know what I'll use them for, it's just nice to have them in reserve.

Share this post


Link to post
Share on other sites
12 hours ago, warspeech said:

For me the focus was on essence removal scrolls - I came out with enough to see me through the rest of my playing time, or the end of the game, whichever comes first.  Like you I also got a lot of tomes, marks, and meds from the daily present sacks for a trivial amount of time.  Not that I know what I'll use them for, it's just nice to have them in reserve.

Same here, Warspeech. I now have a stack of essence removal scrolls that I can use at my leisure...or sell....or ignore. Of course, it would help if the game was actually playable for me. As it is now, I tend to ride 3-10 virtual feet, freeze, ride another 3-10 virtual feet, freeze again, etc.

Turbine never fails to disappoint.

Share this post


Link to post
Share on other sites

I feel sincerely sad for the tech people at Turbine who's trying to fix this DEBACLE while their efforts are being marred by the usual BULLSHIT Turbine's PR are throwing at their customers. "We are continuing to look into the connection issues"... "We are continuing to work towards putting out a hotfix to address the issues"... "We do not yet have a time for the hotfix but we will post it as soon as we know." "We are testing a potential fix to the recent server issues." "Citizens please open a ticket and stand in line"... "There's nothing wrong with you radio yadda yadda yadda"... And then there's the legion of FANBOIS screaming "Well done Frelorn" and "Thanks Vyv" and ignoring how AWFUL Turbine has always been whenever they need to perform key tasks (anyone remember Codemasters' migration?) finishing with the release of MT and the dismal Bullroarer tests on the new, improved hardware.

"New" "improved" hardware. They're asking for a leap of faith in order for their customers to swallow this toad. While they don't release tech docs (and they won't), I'll translate "new" as "not the ones we were using" and "improved" as "they allow us to save lots of money yippee".

I think that Turbine won't be fixing this as there's no fix. The fix would be more powerful hardware which they cannot / will not afford. The management is just waiting for people to get used to the higher latencies and the higher lag - most of us will, and we will forget that the game used to run smoother. Some people will be leaving of course - it's all a matter of how many. If enough remain in order for the operation to be financially productive, management will probably get a bonus. If too many leave, they'll just pull the plug. They're just raising the ante. Well, getting to Mordor was not supposed to be a Casual Stroll, was it?

Sorry for the rant, I had to let steam off and won't risk doing so at their toxic forums.

  • Upvote 1

Share this post


Link to post
Share on other sites
On 1/12/2016 at 9:34 PM, nosam9 said:

Something went wrong. Apparently the game is barely playable.  They wont get it working until tomorrow.

FREELORN posted:

"We believe we have found a fix for the current login and latency issues that players are reporting with the game. However a hotfix to the game will be required. In order for us to do this we will have to build an update, deploy it to our test server, and test it extensively before we can release the fix to the live servers. This means that while we will continue to try and mitigate any issues that occur tonight, we will not be able to deploy a fix for this issue until sometime tomorrow.

We do understand this is not an optimal solution, however right now this is the only option we have. This issue was something that for whatever reason, did not appear on any of our tests including the live testing we did on Bullroarer. We apologize for any inconvenience this may cause players."

What bullshit.

 

If you are already 100% down, that is the one time where it is appropriate to deploy an untested fix. It can't make it worse and if it works you are better off than before.

 

Unless of course this means "we have no idea what is going on and the testing I am talking about is to find out what's wrong to maybe develop a patch".

 

Also, how the heck, anyway? Presumably it can't be a 64 bit issue since they presumably tested the 64 bit code before deployment. What in the world are they changing that it is so difficult? And couldn't have been tested before switching the customer base?

Share this post


Link to post
Share on other sites
1 hour ago, MusicMachine said:

...

I think that Turbine won't be fixing this as there's no fix. The fix would be more powerful hardware which they cannot / will not afford.

 

Good post. 

I think they are majorly screwed at this point. I would guess the best case scenario is they get the major problems fixed early next week. That will not go over well with players who expect to actually be able to play an MMO with minimal down time.

Reading that thread over there, it looks really bad. And they have set up the expectation that the new servers will improve things - but I am not so sure about that. It still looks like they have done every as cheaply as possible, including the new servers. Plus, what do you expect when you have people with no expertise trying to do the server transfer. I applaud Freelorn for staying up until 2:00 am, but it seems like they have hardly anyone with the technical background working on this. Probably just their regular devs trying to figure the server stuff out.  

It's gonna be ugly. I would not be at all surprised if this is the final blow to the game. How much money is Turbine really willing to spend to fix this? 

Share this post


Link to post
Share on other sites

I don't think that lack of CPU power was the root of any of their technical problems since U6 (when some of the lag became noticeable in one step).

Share this post


Link to post
Share on other sites
5 minutes ago, nosam9 said:

It's gonna be ugly. I would not be at all surprised if this is the final blow to the game. How much money is Turbine really willing to spend to fix this? 

If Turbine was paying, I think they'd do it. But it's WB who's holding the checkbook, and their business is not exactly charities.

3 minutes ago, Darmokk said:

I don't think that lack of CPU power was the root of any of their technical problems since U6 (when some of the lag became noticeable in one step).

You're probably right, but less horsepower would not help either XD. Although they have swapped systems from 32 to 64 bit, I don't really think that they have significantly improved hardware in order to benefit from the move.

Come to think of it, most companies would outsource a migration like this. Maybe it's not Turbine's crew who's turning the gears.

Share this post


Link to post
Share on other sites
25 minutes ago, MusicMachine said:

If Turbine was paying, I think they'd do it. But it's WB who's holding the checkbook, and their business is not exactly charities.

We don't know exactly how much WB are making the decisions. But to me at least it is very clear Turbine management has planned not spending much on Lotro since F2P. It looks again to me that they decided to spend as little as possible on the new servers. So I would expect they did this transfer without qualified staff, to save money, and they will not like the idea of bringing in any experts to sort it out at additional cost.  It always seems to be: spend as little as possible on Lotro but keep the game running so money can come in from the store.  

If they wanted to do this right, they would spend a decent amount of money to get Lotro up and running again this week. But I am pretty sure they wont do that, and the people without experience in this area will bumble along hoping they can fix it.

Share this post


Link to post
Share on other sites
2 minutes ago, nosam9 said:

If they wanted to do this right, they would spend a decent amount of money to get Lotro up and running again this week. But I am pretty sure they wont do that, and the people without experience in this area will bumble along hoping they can fix it.

They better hurry or they'll be saving from another round of layoffs. The "official" thread is not getting any better.

Share this post


Link to post
Share on other sites

I wonder if something changed in the overall plan when everyone came back from the holiday.  It almost sounds like they're reacting to a new situation that they didn't know about at the end of the year.  Some new decision from the higher-ups, maybe (regarding the EU datacenter, I mean).

That seems like a separate thing from the overall disaster that's the current state of the game.  That one seems to be business as usual for this crew. Something major got overlooked technically, and now they've gone silent after the flurry of posts earlier this week.  Doesn't look like they'll sort it out anytime soon, either.

Share this post


Link to post
Share on other sites

My guess is that they're in over their heads with a code base no one really knows dealing with a technical situation that just went sideways for everything earlier this month.

I suspect their reason for withholding the EU transfer is because they don't want to add even more problems to a situation that's rapidly degrading.

Honestly, I'm not surprised at this point, given how much talent has been bled from LotRO.  I mean, we should have all realized this when RockX and the other devs left, as the game was forced into maintenance mode at that point in time, and the skeleton crew has been able to tread water thus far.

However, even the best can't tread water forever, as eventually you either give into fatigue, or Jaws gets ya.

Share this post


Link to post
Share on other sites

I was surprised when they announced the data center launch.  I think there was a week from announcement to roll out.  I think what happened was, just like u17,  someone, somewhere said "shit or get off the pot."  If we are to believe that BR was on the new hardware since the first load test they did, then that was quite a while to have two "data centers" open at the same time.  They suddenly reached a point where they were told they needed to make the migration now.  In typical Turbine fashion it's the "release now and worry about fixes later." instead of ensuring everything will go smoothly.

Share this post


Link to post
Share on other sites

Well, they'd almost have to have both datacenters open for a time. The problems and delays they encountered prior to the move likely backed them against the move-out date in Boston, so they probably did move faster than they would have liked.
 

Any move like this was going to have issues, there is simply no way to test everything without it being live. Turbine's issue is that it will take them much longer to fix things than most companies would take. A week or two to shake out the bugs would be expected anywhere, with Turbine, I'm guessing a month and even then something will remain worse than it was before the move. 

 

I did log on late last night. Loading Screens seemed faster and at first the game was very smooth, much better than I've seen it in a while. About 10 min in though I started getting continuous mini rubberbanding that was more frequent and annoying from anything I remember in LotRO before.

Share this post


Link to post
Share on other sites

Does the fact that the datacenter is in NJ, and the team is based in Boston, make it worse in terms of getting things fixed?  Can they do everything they need to do remotely?  (Serious question, I have no idea how setups like that work).

Share this post


Link to post
Share on other sites
1 minute ago, warspeech said:

Does the fact that the datacenter is in NJ, and the team is based in Boston, make it worse in terms of getting things fixed?  Can they do everything they need to do remotely?  (Serious question, I have no idea how setups like that work).

Assuming Windows here, they should be able to RDP into the machines and work from anywhere.  With various PowerShell and other deployment scripts, they can move bits around as expected.  At the end of the day, we're talking about an executable running on a server accessing a database, with enough redundancy built into the design so that if a DB server goes down, the entire cluster that makes up Landroval (for example) doesn't die with the dead DB, likewise for the login server and world server (as I would expect both servers to be on different computers).

If Turbine REALLY wanted to earn some points with the community, they could stick most of the client source (sans graphical assets and keys) and build scripts up on GitHub, and let us fix their shit for them.

Share this post


Link to post
Share on other sites

uw =)

If you want a better insight into what's going on, you may want to read http://community.eveonline.com/news/dev-blogs/tranquility-tech-3/?_ga=1.33122046.156964246.1435699359  Be warned that you may need Wikipedia open to translate the tech, as it's an article written for someone that understands most of the acronym soup of modern computing.

Share this post


Link to post
Share on other sites
4 minutes ago, Almagnus1 said:

uw =)

If you want a better insight into what's going on, you may want to read http://community.eveonline.com/news/dev-blogs/tranquility-tech-3/?_ga=1.33122046.156964246.1435699359  Be warned that you may need Wikipedia open to translate the tech, as it's an article written for someone that understands most of the acronym soup of modern computing.

I remember reading that article when someone linked to it at the official forums and going glow in the dark green with envy. Made me want to play EVE Online even if I'm really bad at space sims. It still does!

Yup, that's how it should be done, both in systems and communication - but I have the distinct feeling that Turbine's datacenter will not match the described hardware, not by a long shot. Wish they proved me wrong. And about communications... we will NEVER see anything as informative from Turbine. They won't release the tiniest piece of info about their current or former systems. The only thing they can currently do is average corporate blurb.

Share this post


Link to post
Share on other sites
6 hours ago, nosam9 said:

It's gonna be ugly. I would not be at all surprised if this is the final blow to the game. How much money is Turbine really willing to spend to fix this? 

Time to reset the nosam9 death of LotRO counter.

Share this post


Link to post
Share on other sites
1 hour ago, Spiteful said:

LOL

Someday it will close down and then you'll see he was right!

If Turbine can't get their shit together within the next month or so, it's gonna be sooner than later....

Share this post


Link to post
Share on other sites
3 hours ago, Almagnus1 said:

If Turbine REALLY wanted to earn some points with the community, they could stick most of the client source (sans graphical assets and keys) and build scripts up on GitHub, and let us fix their shit for them.

Allowing the players to fix stuff couldn't be any worse than Turbine doing it.  I'm sure there are plenty of smart people playing lotro, at least what some of them keeping telling everyone. :)

Share this post


Link to post
Share on other sites
55 minutes ago, Almagnus1 said:

If Turbine can't get their shit together within the next month or so, it's gonna be sooner than later....

Yea but after saying the same thing for multiple years over and over.... I mean hell, even a broken clock is eventually right.

And honestly, I don't see LotRO shutting down for at least another year, and if the licence can be renewed, it's going to keep going, despite if some people think it should or not, because enough people think it should and pay.

 

Besides a Nosam9 Death of LotRO Counter just struck me as funny.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×